Image generation device, image generation method, recording medium, and method for generating an in-focus image based on feature points

ABSTRACT

An image generation device generates an image of an object placed on a surface of an image sensor by using a plurality of images each of which is captured by the image sensor when the object is irradiated with a corresponding one of a plurality of illuminators. The object includes first object and one or more second objects included in the first object. The image generation device determines a section of the first object including a largest number of feature points of second objects, generates an in-focus image using the section as a virtual focal plane, and causes the in-focus image to be displayed on a display screen.

BACKGROUND

1. Technical Field

The present disclosure relates to a technique for use in lenslessmicroscopes to generate an image of an object by using a virtual focalplane on the basis of a plurality of images captured by using aplurality of light sources.

2. Description of the Related Art

There is a need for continuous observation of cultured cells withoutstaining the cells in many fields in which cultured cells are used formedical and industrial purposes, such as production of cells for use inmedical treatment and investigation of the efficacy of a medicine.However, since most cells are colorless and transparent, thethree-dimensional structure of cultured cells is not clearly revealed byimaging with optical microscopes using transmitted light.

Japanese Unexamined Patent Application Publication No. 2013-101512discloses a method for generating an in-focus image (virtual sectionalimage) at a plane that is not parallel to an objective lens from manyimages for which the focal plane is parallel to the objective lens andthe focal point is at different heights with respect to an object (i.e.,many images captured by changing the focus along a height direction ofthe object) in order to evaluate the sectional profile of cells.

Continuous observation of cultured cells is carried out in a limitedspace, such as an incubator, in order to maintain a humid environmentfor culturing the cells. To enable observation in a limited humid space,U.S. Patent Application Publication No. 2014/0133702 discloses alensless microscope that enables observation of minute cells withoutusing lenses. U.S. Patent Application Publication No. 2014/0133702discloses a method for increasing the resolution by superimposing aplurality of images captured under illumination from a plurality ofdifferent positions (ptychography).

According to the method disclosed in Japanese Unexamined PatentApplication Publication No. 2013-101512, since a partial image isextracted from each image at a corresponding one of the heights afterimaging and then the extracted partial images are linked together, thejoints of the partial images become discontinuous. Consequently, theimage quality of the virtual sectional image degrades due todiscontinuity. In addition, if processing for blurring is performed onthe discontinuous portions to decrease the degradation of image qualitydue to discontinuity, the sharpness of the virtual sectional imagedecreases.

SUMMARY

One non-limiting and exemplary embodiment provides an image generationdevice and the like capable of generating an image of an object by usinghigh-quality in-focus images that are generated for respective virtualfocal planes by using a plurality of captured images.

In one general aspect, the techniques disclosed here feature an imagegeneration device including a plurality of illuminators; an image sensorhaving a surface on which an object is placed; and a control circuitthat generates an in-focus image of the object at a virtual focal planelocated between the image sensor and the plurality of illuminators,wherein the object includes a first object and one or more secondobjects included in the first object, and wherein the control circuit(a1) obtains a plurality of images captured by the image sensor, each ofthe plurality of images being captured when a corresponding one of theplurality of illuminators irradiates the object with light, (a2)identifies feature points of the one or more second objects included ineach of the plurality of images that have been obtained, (a3) calculatesthree-dimensional positions of the feature points of the one or moresecond objects on the basis of positions of the feature points of theone or more second objects in each of the plurality of images andpositions of the plurality of illuminators, and (a4) determines asection of the first object including a largest number of feature pointsof second objects among the one or more second objects, generates anin-focus image of the section, and causes the in-focus image of thesection to be displayed on a display screen.

According to embodiments of the present disclosure, an image of anobject is successfully generated by using high-quality in-focus imagesthat are generated for respective virtual focal planes by using aplurality of captured images.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a computer-readable recording medium, or any selectivecombination of a device, a system, a method, an integrated circuit, acomputer program, and a computer-readable recording medium. Examples ofthe computer-readable recording medium include nonvolatile recordingmedia, such as a Compact Disc-Read Only Memory (CD-ROM).

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the functionalconfiguration of an image generation system according to a firstembodiment;

FIG. 2 is a diagram schematically illustrating an example of thestructure of an illuminator according to the first embodiment;

FIG. 3 is a diagram schematically illustrating an example of thestructure of the illuminators according to the first embodiment;

FIG. 4 is a diagram schematically illustrating an example of thestructure of the illuminators according to the first embodiment;

FIG. 5 is a schematic diagram for describing a condition of the diameterto be met by a pinhole of the illuminator according to the firstembodiment;

FIG. 6 is a diagram illustrating an example of information stored in astorage unit according to the first embodiment;

FIG. 7 is a flowchart illustrating an example of an operation of theimage generation system according to the first embodiment;

FIG. 8 is a schematic diagram illustrating an example of a relationshipbetween coordinates and a focal plane;

FIG. 9 is a flowchart illustrating an example of an operation of animaging device according to the first embodiment;

FIG. 10 is a flowchart illustrating an example of an operation of arefocusing processing unit according to the first embodiment;

FIG. 11 is a schematic diagram for describing a specific example of arefocusing process according to the first embodiment;

FIG. 12 is a schematic diagram for describing the specific example ofthe refocusing process according to the first embodiment;

FIG. 13 is a schematic diagram for describing the specific example ofthe refocusing process according to the first embodiment;

FIG. 14 is a schematic diagram for describing the specific example ofthe refocusing process according to the first embodiment;

FIG. 15 is a schematic diagram for describing the specific example ofthe refocusing process according to the first embodiment;

FIG. 16 is a flowchart illustrating an example of an operation of therefocusing processing unit according to the first embodiment;

FIG. 17 is a perspective view of an embryo which is an example of anobject to be imaged;

FIG. 18 is a schematic diagram of a stack of reference in-focus imagesat a plurality of reference focal planes that are stacked in a line inthe arrangement order;

FIG. 19 is a diagram illustrating an example in which the plurality ofreference in-focus images are displayed using a photograph;

FIG. 20 is a diagram illustrating a stack of a plurality ofbackground-removed in-focus images that are stacked in a line in thearrangement order of respective reference focal planes;

FIG. 21 is a diagram illustrating an example in which the plurality ofbackground-removed in-focus images are displayed using a photograph;

FIG. 22 is a schematic diagram illustrating an example of an outlinereference in-focus image;

FIG. 23 is a schematic diagram illustrating an example of athree-dimensional (3D) outline of an embryo;

FIG. 24 is a diagram illustrating an example of a plurality of referencesectional images;

FIG. 25 is a diagram illustrating an example in which the plurality ofreference sectional images are displayed using a photograph;

FIG. 26A is a diagram illustrating an example of a 3D model of theembryo displayed on a display screen of a display unit;

FIG. 26B is a diagram illustrating an example of the 3D model of theembryo displayed on the display screen of the display unit;

FIG. 26C is a diagram illustrating an example of the 3D model of theembryo displayed on the display screen of the display unit;

FIG. 27 is a diagram illustrating a screen in which part of the exampledisplayed on the display screen in FIG. 26C is displayed using aphotograph;

FIG. 28 is a flowchart illustrating an example of an operation performedby the image generation system according to the first embodiment todisplay a detailed sectional image of the embryo;

FIG. 29 is a diagram illustrating an example case where a pixel value ofeach pixel on a section of the embryo to be displayed in detail isestimated by using the corresponding pixels of reference sectionalimages at two reference focal planes that cross the section;

FIG. 30 is a diagram illustrating a relationship between a pixel on asection of the embryo to be displayed in detail and the correspondingpixels of the two reference sectional images, the relationship beingused to calculate the pixel value of the pixel;

FIG. 31 is a block diagram illustrating an example of the functionalconfiguration of an image generation system according to a secondembodiment;

FIG. 32 is a diagram illustrating an example of early embryo modelsstored in a first memory;

FIG. 33 is a diagram illustrating an example of an optimum sectionsetting table stored in the first memory;

FIG. 34 is a flowchart illustrating an example of an operation performedby the image generation system according to the second embodiment todisplay an optimum sectional image;

FIG. 35 is a plan view of a plurality of illuminators viewed in adirection from the illuminators toward an image sensor;

FIG. 36 is a diagram illustrating a correspondence of each identicalelement in a plurality of captured images belonging to a group ofcaptured images;

FIG. 37 is a diagram illustrating the positions of the center of eachidentical element in the plurality of captured images;

FIG. 38 is a diagram illustrating a positional relationship among thecenters of each identical element in the plurality of captured images;

FIG. 39 is a diagram illustrating a positional relationship among thecenters of the identical element in the plurality of photographic imagesand illuminators;

FIG. 40 is a diagram illustrating a positional relationship between thecenters of two cells of the embryo;

FIG. 41 illustrates an example in which the optimum sectional image ofthe embryo is displayed on a display screen of a display unit;

FIG. 42 is a diagram illustrating an example in which the optimumsectional image of the embryo is displayed using a photograph;

FIG. 43 is a diagram illustrating an example of a focal plane selectionscreen displayed on the display screen of the display unit;

FIG. 44 is a diagram illustrating an example in which the focal planeselection screen is displayed using a photograph;

FIG. 45 is a flowchart illustrating an example of an operation of therefocusing processing unit according to a first modification of thefirst and second embodiments;

FIG. 46 is a flowchart illustrating an example of an operation of therefocusing processing unit according to a second modification of thefirst and second embodiments;

FIG. 47 is a schematic diagram for describing a specific example of arefocusing process according to the second modification of the first andsecond embodiments;

FIG. 48 is a schematic diagram for describing the specific example ofthe refocusing process according to the second modification of the firstand second embodiments;

FIG. 49 is a schematic diagram for describing the specific example ofthe refocusing process according to the second modification of the firstand second embodiments;

FIG. 50 is a schematic diagram for describing the specific example ofthe refocusing process according to the second modification of the firstand second embodiments;

FIG. 51 is a block diagram illustrating an example of the functionalconfiguration of an image generation system according to a thirdmodification of the first and second embodiments;

FIG. 52 is a diagram schematically illustrating an example of a range ofthe illumination position according to the third modification of thesecond embodiment;

FIG. 53 is a schematic diagram in which a relationship between the focallength of a lens and the depth of field is associated with arelationship between arrangement of a point light source at the time ofrefocusing and the depth of field; and

FIG. 54 is a flowchart illustrating an example of an operation of theimage generation system according to the third modification of thesecond embodiment.

DETAILED DESCRIPTION

An image generation device according to another aspect of the presentdisclosure includes a plurality of illuminators; an image sensorincluding a plurality of sensor pixels, the image sensor having asurface on which an object is placed; and a control circuit thatgenerates a plurality of reference in-focus images each corresponding toone of a plurality of virtual reference focal planes that are locatedbetween the image sensor and the plurality of illuminators and generatesa three-dimensional image of the object by using the plurality ofreference in-focus images, wherein the image sensor captures a pluralityof images, each of the plurality of images being captured by using pixelvalues based on light received by the plurality of sensor pixels when acorresponding one of the plurality of illuminators irradiates the objectwith the light, wherein each of the plurality of reference in-focusimages includes a plurality of in-focus pixels, and wherein the controlcircuit (a1) obtains the plurality of images captured by the imagesensor, (a2) obtains information regarding the plurality of virtualreference focal planes that pass through the object and are spaced apartfrom one another, (a3) generates the plurality of reference in-focusimages by obtaining pixel values of the sensor pixels corresponding tothe plurality of in-focus pixels of the plurality of reference in-focusimages by using the information regarding the plurality of virtualreference focal planes and the plurality of images, (a4) extracts anoutline of the object by using a reference in-focus image including anoutline of the object having the highest contrast from among theplurality of reference in-focus images, (a5) identifies athree-dimensional outline of the object on the basis of the extractedoutline of the object, (a6) generates a plurality of reference sectionalimages of the object by removing a region outside the three-dimensionaloutline from the plurality of reference in-focus images, and (a7)generates the three-dimensional image of the object by using theplurality of reference sectional images and causes the three-dimensionalimage of the object to be displayed on a display screen.

According to this aspect, a plurality of in-focus images (i.e.,reference in-focus images) are successfully generated for a plurality offocal planes that pass through an object, and a three-dimensional (3D)image of the object is successfully generated by using the generatedin-focus images. The use of in-focus images for a plurality of focalplanes that pass through an object enables a 3D image of the object tobe displayed three-dimensionally including contents of the object evenif the object is translucent or transparent. In addition, since thein-focus images are generated for the plurality of focal planes insteadof the entire region of the object, a processing amount required forgeneration of the 3D image of the object is successfully reduced. Notethat the surface of the image sensor on which the object is placedincludes a surface above sensor pixels of the image sensor.

In the image generation device according to the other aspect of thepresent disclosure, the control circuit may select a section of theobject in the displayed three-dimensional image of the object inaccordance with an instruction externally input, may generate an imageof the selected section of the object by using pixel values of aplurality of pixels of the plurality of reference in-focus images, theimage of the selected section of the object including a plurality ofsection pixels, and may calculate a pixel value of each of the pluralityof section pixels of the image of the selected section of the object byusing a pixel value of a pixel of the reference in-focus image locatedat the section pixel or by using pixel values of pixels of the referencein-focus images located near the section pixel.

According to this aspect, a given section is successfully selected byusing the 3D image of the object and an image of the selected section issuccessfully displayed. Since pixel values of a plurality of sectionpixels of the sectional image of the object are calculated by usingpixel values of the respective pixels of the reference in-focus imagethat are located at the section pixels or pixel values of pixels of thereference in-focus images near the respective section pixels, thesectional image of the object can be a sharp image in whichdiscontinuity and blur are reduced.

In the image generation device according to the other aspect of thepresent disclosure, the pixels of the reference in-focus images locatednear the section pixel that are used to calculate the pixel value of thesection pixel may be pixels of the reference in-focus images for twovirtual reference focal planes having the section pixel interposedtherebetween. According to this aspect, since a pixel value of eachsection pixel is calculated by using pixels of reference in-focus imagesat respective reference focal planes located on the respective sides ofthe section pixel, the pixel value can be highly accurate.

In the image generation device according to the other aspect of thepresent disclosure, the control circuit may generate a preview sectionalimage representing a section of the object for preview and cause thepreview sectional image to be displayed on the display screen, thepreview sectional image including a plurality of pixels, and the controlcircuit may generate the preview sectional image by using, as a pixelvalue of each of the plurality of pixels of the preview sectional image,a pixel value of a pixel of the reference in-focus image located at thepixel of the preview sectional image. According to this aspect, the useris allowed to select a to-be-displayed section of the object withreference to a preview sectional image. In addition, since a pixel valueof each pixel of the reference in-focus image is used as a pixel valueof a corresponding pixel of the preview sectional image withoutprocessing the pixel value, the preview sectional image is generatedeasily.

In the image generation device according to the other aspect of thepresent disclosure, the control circuit may calculate the pixel value ofeach of the plurality of in-focus pixels by using a pixel value of eachof the sensor pixels that satisfy a relationship in which the positionof the illuminator, the position of the in-focus pixel, and the positionof the sensor pixel are on a line. According to this aspect, for eachpixel of an in-focus image at the focal plane, pixel values of theplurality of photographic images corresponding to the pixel can bereflected. Thus, a high-quality in-focus image of the object issuccessfully generated.

In the image generation device according to the other aspect of thepresent disclosure, the object may be an embryo, the outline of theembryo included in the reference in-focus image may be circular, and thethree-dimensional outline of the embryo may be spherical. In thisaspect, cells included in an embryo are seen through from outside theembryo. The image sensor can capture images of the embryo and cellsirradiated with light by respective illuminators. An embryo having suchproperties is suitably used for image generation performed by the imagegeneration device.

An image generation method according to another aspect of the presentdisclosure is an image generation method for generating an image of anobject placed on an image sensor, including (b1) capturing a pluralityof images, each of the plurality of images being captured by using pixelvalues based on light received by a plurality of sensor pixels of theimage sensor when a corresponding one of a plurality of illuminatorsirradiates the object with the light; (b2) setting a plurality ofvirtual reference focal planes between the image sensor and theplurality of illuminators, the plurality of virtual reference focalplanes passing through the object and being spaced apart from oneanother; (b3) generating a plurality of reference in-focus images eachcorresponding to one of the plurality of virtual reference focal planesby obtaining pixel values of the sensor pixels corresponding to aplurality of in-focus pixels of the plurality of reference in-focusimages by using information regarding the plurality of virtual referencefocal planes and the plurality of captured images; (b4) extracting anoutline of the object by using a reference in-focus image including anoutline of the object having the highest contrast from among theplurality of reference in-focus images; (b5) identifies athree-dimensional outline of the object on the basis of the extractedoutline of the object; (b6) generating a plurality of referencesectional images of the object by removing a region outside thethree-dimensional outline of the object from the plurality of referencein-focus images; and (b7) generating a three-dimensional image of theobject by using the plurality of reference sectional images and causingthe three-dimensional image of the object to be displayed on a displayscreen, at least one of (b1) to (b7) being performed by a controlcircuit.

The image generation method according to the other aspect of the presentdisclosure may further include (c1) selecting a section of the object inthe three-dimensional image of the object; (c2) generating an image ofthe selected section of the object by using pixel values of a pluralityof pixels of the plurality of reference in-focus images, the image ofthe selected section of the object including a plurality of sectionpixels; and (c3) calculating a pixel value of each of the plurality ofsection pixels of the image of the selected section of the object byusing a pixel value of a pixel of the reference in-focus image locatedat the section pixel or by using pixel values of pixels of the referencein-focus images located near the section pixel.

Further, in the image generation method according to the other aspect ofthe present disclosure, in the calculating of a pixel value of each ofthe plurality of section pixels, pixels of the reference in-focus imagesfor two virtual reference focal planes having the section pixelinterposed therebetween may be used as the pixels of the referencein-focus images located near the section pixel.

The image generation method according to the other aspect of the presentdisclosure may further include (d1) generating a preview sectional imagerepresenting a section of the object for preview and causing the previewsectional image to be displayed on the display screen, the previewsectional image including a plurality of pixels, and in the generatingof the preview sectional image, as a pixel value of each of theplurality of pixels of the preview sectional image, a pixel value of apixel of the reference in-focus image located at the pixel of thepreview sectional image may be used.

In the image generation method according to the other aspect of thepresent disclosure, the pixel value of each of the plurality of in-focuspixels may be calculated by using a pixel value of each of the sensorpixels that satisfy a relationship in which the position of theilluminator, the position of the in-focus pixel, and the position of thesensor pixel are on a line.

In the image generation device according to the other aspect of thepresent disclosure, the object may be an embryo, the outline of theembryo included in the reference in-focus image may be circular, and thethree-dimensional outline of the embryo may be spherical.

A recording medium according to another aspect of the present disclosureis a recording medium storing a control program that causes a deviceincluding a processor to perform a process, the recording medium beingnonvolatile and computer-readable, the process including (e1) capturing,using an image sensor, a plurality of images of an object placed on theimage sensor, each of the plurality of images being captured by usingpixel values based on light received by a plurality of sensor pixels ofthe image sensor when a corresponding one of a plurality of illuminatorsirradiates the object with the light; (e2) setting a plurality ofvirtual reference focal planes between the image sensor and theplurality of illuminators, the plurality of virtual reference focalplanes passing through the object and being spaced apart from oneanother between the image sensor and the plurality of illuminators; (e3)generating a plurality of reference in-focus images each correspondingto one of the plurality of virtual reference focal planes by obtainingpixel values of the sensor pixels corresponding to a plurality ofin-focus pixels of the plurality of reference in-focus images by usinginformation regarding the plurality of virtual reference focal planesand the plurality of captured images; (e4) extracting an outline of theobject by using a reference in-focus image including an outline of theobject having the highest contrast from among the plurality of referencein-focus images; (e5) identifying a three-dimensional outline of theobject on the basis of the extracted outline of the object; (e6)generating a plurality of reference sectional images of the object byremoving a region outside the three-dimensional outline of the objectfrom the plurality of reference in-focus images; and (e7) generating athree-dimensional image of the object by using the plurality ofreference sectional images and causing the three-dimensional image ofthe object to be displayed on a display screen.

An image generation device according to an aspect of the presentdisclosure includes a plurality of illuminators; an image sensor havinga surface on which an object is placed; and a control circuit thatgenerates an in-focus image of the object at a virtual focal planelocated between the image sensor and the plurality of illuminators,wherein the object includes a first object and one or more secondobjects included in the first object, and wherein the control circuit(a1) obtains a plurality of images captured by the image sensor, each ofthe plurality of images being captured when a corresponding one of theplurality of illuminators irradiates the object with light, (a2)identifies feature points of the one or more second objects included ineach of the plurality of images that have been obtained, (a3) calculatesthree-dimensional positions of the feature points of the one or moresecond objects on the basis of positions of the feature points of theone or more second objects in each of the plurality of images andpositions of the plurality of illuminators, and (a4) determines asection of the first object including a largest number of feature pointsof second objects among the one or more second objects, generates anin-focus image of the section, and causes the in-focus image of thesection to be displayed on a display screen.

According to this aspect, the image generation device selects a sectionincluding a largest number of feature points of the second object(s)included in the first object and displays an in-focus image of theselected section. The displayed in-focus image of the sectionsuccessfully shows many features inside the first object. Accordingly,the image generation device is capable of automatically generating andproviding useful information to the user.

In the image generation device according to the aspect of the presentdisclosure, the control circuit may associate with each other thefeature points of each of the one or more second objects in theplurality of images when the feature point of the second object isidentified. According to this aspect, corresponding feature points ofthe second object are identified in captured images. For example, whentwo or more feature points are set, a positional relationship betweencorresponding feature points in captured images can be calculated.Consequently, when the first object includes two or more second objects,the image generation device can include more second objects in anin-focus image of the section by associating corresponding featurepoints of each of the second objects.

In the image generation device according to the aspect of the presentdisclosure, the first object may be a spherical embryo, the secondobject may be a cell, and the feature point may be a center point of thecell. In this aspect, cells included in an embryo are seen through fromoutside the embryo. The image sensor can capture images of the embryoand cells irradiated with light by respective illuminators. An embryohaving such properties is suitably used for image generation performedby the image generation device.

An image generation method according to an aspect of the presentdisclosure is an image generation method for generating an image of anobject placed on an image sensor by using an in-focus image at a virtualfocal plane located between the image sensor and a plurality ofilluminators, the object including a first object and one or more secondobjects included in the first object, the image generation methodincluding (b1) obtaining a plurality of images captured by sequentiallycausing the plurality of illuminators to irradiate the object withlight; (b2) identifies feature points of the one or more second objectsincluded in each of the plurality of images that have been obtained;(b3) calculating three-dimensional positions of the feature points ofthe one or more second objects on the basis of positions of the featurepoints of the one or more second objects in the plurality of images andpositions of the plurality of illuminators; and (b4) determining asection of the first object including a largest number of feature pointsof second objects among the one or more second objects, generating anin-focus image of the section, and causing the in-focus image of thesection to be displayed on a display screen, at least one of (b1) to(b4) being performed by a control circuit.

In the image generation method according to the aspect of the presentdisclosure, the feature points of each of the one or more second objectsin the plurality of images may be associated with each other when thefeature point of the second object is identified.

In the image generation method according to the aspect of the presentdisclosure, the first object may be a spherical embryo, the secondobject may be a cell, and the feature point may be a center point of thecell.

A recording medium according to an aspect of the present disclosure is arecording medium storing a control program causing a device including aprocessor to perform a process, the recording medium being nonvolatileand computer-readable, the process including (c1) obtaining a pluralityof images, each of the plurality of images being captured by an imagesensor when a corresponding one of a plurality of illuminatorsirradiates an object with light, the object being placed on the imagesensor and including a first object and one or more second objectsincluded in the first object; (c2) identifying feature points of the oneor more second objects in each of the plurality of images that have beenobtained; (c3) calculating three-dimensional positions of the featurepoints of the one or more second objects on the basis of positions ofthe feature points of the one or more second objects in the plurality ofimages and positions of the plurality of illuminators; and (c4)determining a section of the first object including a largest number offeature points of second objects among the one or more second objects,generating an in-focus image using the section as a virtual focal plane,and causing the in-focus image of the section to be displayed on adisplay screen.

In the recording medium according to the aspect of the presentdisclosure, the feature points of each of the one or more second objectsin the plurality of captured images may be associated with each otherwhen the feature point of the second object is identified.

In addition, in the recording medium according to the aspect of thepresent disclosure, the first object may be a spherical embryo, thesecond object may be a cell, and the feature point may be a center pointof the cell.

It should be noted that general or specific embodiments of these imagegeneration devices and image generation methods may be implemented as adevice, a method, an integrated circuit, a computer program, acomputer-readable recording medium such as a CD-ROM, or any selectivecombination thereof. For example, the image generation methods may beimplemented by a processor such as a central processing unit (CPU) or amicro processing unit (MPU); a circuit such as a large scale integration(LSI) chip; an integrated circuit (IC) card; or a discrete module, orthe like.

In addition, processes according to embodiments may be implemented by asoftware program or digital signals based on the software program. Forexample, the software program and the digital signals based on thesoftware program may be stored on a computer-readable recording medium,for example, a flexible disk, a hard disk, a CD-ROM, an MO, a DigitalVersatile Disc (DVD), a DVD-ROM, DVD-random access memory (RAM), BD(Blu-ray (registered trademark) Disc), or a semiconductor memory. Inaddition, the software program and the digital signals based on thesoftware program may be transmitted via an electrical communicationline, a wireless or wired communication line, a network such as theInternet, data broadcasting, or the like. In addition, the softwareprogram and the digital signals based on the software program may betransferred to another independent computer system after being recordedon a recording medium or via a network or the like and may be executedby the other independent computer system.

Image generation systems according to an aspect of the presentdisclosure will be described below with reference to the drawings. Notethat each embodiment to be described below provides general or specificexamples. The values, shapes, components, arrangement and connection ofthe components, steps, the order of steps, etc., described in thefollowing embodiments are merely illustrative and are not intended tolimit the claims. Among the components in the following embodiments, acomponent not recited in any of the independent claims indicating themost generic concept is described as an optional component. In thefollowing description of the embodiments, the expression accompanying“substantially”, such as substantially parallel or substantiallyorthogonal, is sometimes used. For example, the expression“substantially parallel” not only indicates the state of beingcompletely parallel but also indicates the state of being substantiallyparallel, that is, the state of allowing an error of several percent,for example. The same applies to other expressions accompanying“substantially”.

First Embodiment

An image generation system including an image generation deviceaccording to a first embodiment generates an image of an object at avirtual focal plane located between a plurality of illuminators and animage sensor by using a plurality of images each of which is captured byimaging the object when the object placed on the image sensor isirradiated with light by a corresponding one of the plurality ofilluminators. The image generated by using the plurality of capturedimages is also referred to as an in-focus image or a refocusing image,and generating an image of an object at a virtual focal plane by usingcaptured images is also referred to as a refocusing process. During therefocusing process, pixels on the virtual focal plane may be determinedby using pixels of captured images. The image generation systemgenerates in-focus images at a plurality of virtual focal planes andgenerates a three-dimensional (3D) model of the object by using theplurality of generated in-focus images. Further, the image generationsystem generates a given sectional image of the 3D model by using theplurality of in-focus images included in the 3D model.

1-1. Configuration of Image Generation System

1-1-1. Overall Configuration of Image Generation System

FIG. 1 is a functional block diagram of an image generation system 10according to the first embodiment. The image generation system 10illustrated in FIG. 1 includes an imaging device 100, an imagegeneration device 200, a storage unit 120, and a display unit 150. Theimage generation system 10 may further include a first memory 121 thatstores information regarding predetermined focal planes, the shape of anobject to be imaged, and the like; a second memory 122 that storesinformation regarding pixel(s) that have been subjected to refocusingprocessing; a focal plane input unit 130 that accepts input ofspecifying information for specifying a focal plane; and a computergraphics (CG) operation input unit 140 that accepts input of anoperation instruction for an object displayed on the display unit 150.The display unit 150 is implemented by a display and displays an imageor the like generated by an image operation unit 260 and a sectionalimage generation unit 270. The focal plane input unit 130 and the CGoperation input unit 140 may be implemented by various input devices,such as a keyboard, a mouse, or a touchpad of a computer device or thelike or by an input device based on a screen, such as a touchscreen, ofthe display unit 150.

1-1-2. Configuration of Imaging Device

The configuration of the imaging device 100 will be described first. Theimaging device 100 includes a plurality of illuminators 101, an imagesensor 102, and an imaging control unit 103. The imaging device 100captures images (photographic images) of an object. Note that theimaging device 100 does not include a focus lens.

The object to be imaged includes, for example, a plurality oftranslucent objects placed on the surface of the image sensor 102.Specifically, the object is placed on a plurality of pixels, which aresensor pixels included in the image sensor 102 (described later). Thesurface of the image sensor 102 on which the object is placed includes asurface above the pixels of the image sensor 102. A specific example ofthe object is an early embryo of vertebrate animals, that is, aspherical embryo. A plurality of elements may three-dimensionallyoverlap in the object. A specific example of the plurality of elementsis spherical cells. Herein, an embryo is an example of a first object,and a cell is an example of a second object. The first embodiment willbe described below by using an embryo. The shape of an object to beimaged as the first object is not limited to the sphere, and the objectmay have any shape. For example, the object to be imaged may have anellipsoidal shape, a columnar shape, or a polygonal shape. The shape ofthe plurality of elements each serving as the second object is notlimited to the sphere and may have any shape. For example, the pluralityof elements may have an ellipsoidal shape, a columnar shape, or apolygonal shape. The object to be imaged as the first object may be, forexample, treated to be transparent or translucent so that the pluralityof elements each serving as the second object contained therein is alsoimaged when the object is imaged. The plurality of elements each servingas the second object may be, for example, treated to be transparent ortranslucent so that light from the illuminators 101 passes therethrough;however, the plurality of elements may have a property other than beingtransparent or translucent. The element serving as the second objectneed not be provided in plural and a single element may serve as thesecond object.

The plurality of illuminators 101 are arranged in a line or on asurface, for example. Each of the plurality of illuminators 101 is anilluminator that outputs parallel rays or diffused rays. The pluralityof illuminators 101 include a first illuminator and a secondilluminator. Each of the first and second illuminators radiates raysthat do not cross each other. That is, a plurality of first raysrepresenting first light radiated from the first illuminator do notcross each other. In addition, a plurality of second rays representingsecond light radiated from the second illuminator do not cross eachother. Accordingly, when light is radiated from one of the firstilluminator and the second illuminator, the light from the one of thefirst illuminator and the second illuminator reaches each pixel of theimage sensor 102 from a single direction. That is, light does not reacheach pixel from two or more directions.

Hereinafter, such illumination is referred to as non-crossingillumination. Non-crossing illumination can be implemented by, forexample, parallel rays or diffused rays from a point light source. Theplurality of illuminators 101 sequentially radiate light. The pluralityof illuminators 101 are arranged at different positions and irradiatethe object with light from directions different from one another.

The image sensor 102 includes a plurality of pixels serving as sensorpixels. Each pixel of the image sensor 102 is disposed on alight-receiving surface and obtains intensity of light radiated from theplurality of irradiators 101. The image sensor 102 captures an image onthe basis of intensities of light obtained by the respective pixels.

An example of the image sensor 102 may be a complementary metal-oxidesemiconductor (CMOS) image sensor or a charge coupled device (CCD) imagesensor.

The imaging control unit 103 controls radiation of light performed bythe plurality of illuminators 101 and imaging performed by the imagesensor 102. Specifically, the imaging control unit 103 controls theorder in which the plurality of illuminators 101 radiate light andintervals at which the plurality of illuminators 101 radiate light. Theimaging control unit 103 is constituted by a computer system (notillustrated) including a CPU, a RAM, and a ROM, for example. Functionsof some or all of the components of the imaging control unit 103 may beimplemented as a result of the CPU executing a program stored on the ROMby using the RAM as its work memory. In addition, functions of some orall of the components of the imaging control unit 103 may be implementedby a dedicated hardware circuit.

Light radiated from the plurality of illuminators 101 that are disposedat different positions with respect to the light-receiving surface ofthe image sensor 102 are incident on the light-receiving surface atdifferent incident angles. In the case where the plurality ofilluminators 101 radiate parallel rays, the plurality of illuminators101 radiate parallel rays having different incident angles with respectto the light-receiving surface of the image sensor 102. Parallel rayscan be obtained by diffracting, using a collimating lens 101D, lightemitted from a light-emitting diode (LED) light source 101A via apinhole 101C of a light-shieling plate 101B as illustrated in FIG. 2,for example.

FIG. 3 is a schematic diagram illustrating an example of the structureof the plurality of illuminators 101. In the example of the plurality ofilluminators 101 illustrated in FIG. 3, a plurality of light sources101E each of which radiates parallel rays are fixed at different angleswith respect to the light-receiving surface of the image sensor 102. Inthe example illustrated in FIG. 3, the plurality of light sources 101Eare disposed on the inner surface of a hemisphere 101F that covers theimage sensor 102. Incident angles of light that reaches thelight-receiving surface of the image sensor 102 from the plurality oflight sources 101E are different from one another.

FIG. 4 is a schematic diagram illustrating another example of thestructure of the plurality of illuminators 101. In the example of theplurality of illuminators 101 illustrated in FIG. 4, a plurality ofpseudo point light sources 101G are disposed at different positions on aflat surface 101H that is parallel to the light-receiving surface of theimage sensor 102 so as to face the image sensor 102. Rays emitted fromthe plurality of pseudo point light sources 101G are incident on eachpixel on the light-receiving surface of the image sensor 102 fromdifferent directions. Each of the plurality of pseudo point lightsources 101G is implemented by placing the light-shielding plate 101Bhaving the pinhole 101C near the LED light source 101A, for example. Thediameter of the pinhole 101C is limited by a pixel pitch of the imagesensor 102, a distance between the image sensor 102 and the pinhole101C, and a distance of a point at which an in-focus image is generatedfrom the image sensor 102.

FIG. 5 is a schematic diagram for describing a condition of the diameterto be met by the pinhole 101C. In FIG. 5, d1 denotes the diameter of thepinhole 101C, h1 denotes a distance from the light-receiving surface ofthe image sensor 102 to the pinhole 101C, and h2 denotes a distance fromthe light-receiving surface of the image sensor 102 to a focal point101J (i.e., a point located on a focal plane of a given pixel of anin-focus image). In addition, d2 denotes the diameter of an extent oflight that has passed through the focal point 101J from the pinhole 101and has reached the light-receiving surface of the image sensor 102, andp denotes a pixel pitch of the image sensor 102.

At that time, light that exits from the pinhole 101C ideally passesthrough the focal point 101J and reaches a single point on thelight-receiving surface of the image sensor 102. That is, it isdesirable that light that exits from the pinhole 101C pass through thefocal point 101J and reach a single pixel of the image sensor 102.Accordingly, it is desirable that d2 be smaller than the pixel pitch pof the image sensor 102. That is, d2<p is a condition for realizingnon-crossing illumination as denoted by Equation 1.

$\begin{matrix}{{d\; 2} = {\frac{d\;{1 \cdot h}\; 2}{{h\; 1} - {h\; 2}} < p}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

A condition to be met by d1 can be expressed as Equation 2 by modifyingEquation 1.

$\begin{matrix}{{d\; 1} < \frac{p\left( {{h\; 1} - {h\; 2}} \right)}{h\; 2}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

For example, when the pixel pitch p is equal to 0.001 mm, the distanceh1 from the light-receiving surface of the image sensor 102 to thepinhole 101C is equal to 2 mm, and the distance h2 from thelight-receiving surface of the image sensor 102 to the focal point 101Jis equal to 0.1 mm, the diameter d1 of the pinhole 101C may be smallerthan 0.19 mm.

1-1-3. Configuration of Image Generation Device

The configuration of the image generation device 200 will be describednext. The image generation device 200 is implemented by a controlcircuit. As illustrated in FIG. 1, the image generation device 200includes a focal plane determination unit 210, a refocusing processingunit 220, an in-focus image generation unit 230, a target objectextraction unit 240, a 3D image generation unit 250, the image operationunit 260, and the sectional image generation unit 270.

The focal plane determination unit 210 is implemented by, for example, acontrol circuit or a processor. The focal plane determination unit 210determines a virtual focal plane located between the image sensor 102and the plurality of illuminators 101. Specifically, the focal planedetermination unit 210 determines a focal plane on the basis ofinformation regarding predetermined focal planes stored in the firstmemory 121, for example. The focal plane determination unit 210 may alsodetermine the focal plane in accordance with information input fromoutside via the focal plane input unit 130. In the first embodiment, thefocal plane determination unit 210 determines a plurality of focalplanes that are substantially parallel to the light-receiving surface ofthe image sensor 102. In other words, the focal planes are flat planesin the first embodiment.

The storage unit 120 is implemented by, for example, a semiconductormemory or a hard disk drive. The storage unit 120 stores each imagecaptured by the image sensor 102 together with position information ofthe illuminator 101 used for the imaging.

FIG. 6 illustrates an example of information stored in the storage unit120. The storage unit 120 stores each image file of an image captured bythe imaging device 100 in association with corresponding positioninformation of the illuminator 101 used when the image file was created.In the example illustrated in FIG. 6, position information of theilluminator 101 represents a relative position of the illuminator 101with respect to the image sensor 102. Hereinafter, position informationof the illuminator 101 is also referred to as illumination positioninformation. The illumination position information is stored togetherwith the file ID of each image file and is associated with image datausing the file ID. Note that the illumination position information maybe stored as part of the image file (e.g., header information).

Referring to FIG. 1, the refocusing processing unit 220 is implementedby, for example, a control circuit or a processor. The refocusingprocessing unit 220 calculates intensity of light at each pixel of anin-focus image at a virtual focal plane by using the plurality ofimages, position information of the plurality of illuminators 101, andinformation regarding the virtual focal plane. In the first embodiment,the refocusing processing unit 220 calculates intensities of light atrespective pixels of in-focus images at a plurality of focal planes.Details about this refocusing process will be described later.

The in-focus image generation unit 230 is implemented by, for example, acontrol circuit or a processor. The in-focus image generation unit 230generates an in-focus image at each focal plane from pixel values ofrespective pixels calculated by the refocusing processing unit 220. Apixel value shows brightness of a region in an image.

The target object extraction unit 240 is implemented by, for example, acontrol circuit or a processor. The target object extraction unit 240identifies an outline of an object-showing region, which is a region ofthe object, in an in-focus image at each focal plane and removesbackground that is located outside the outline from the in-focus image.That is, the target object extraction unit 240 generates abackground-removed in-focus image.

The 3D image generation unit 250 is implemented by, for example, acontrol circuit or a processor. The 3D image generation unit 250extracts an outline reference in-focus image which is an image includingthe outline of the object-showing region having the highest contrastfrom among a plurality of background-removed in-focus images. The 3Dimage generation unit 250 identifies a specific 3D outline of the objectin accordance with the shape of two-dimensional (2D) outline in theoutline reference in-focus image and the shape of the object stored inthe first memory 121. The 3D image generation unit 250 furtherassociates the plurality of background-removed in-focus images with the3D outline of the object and removes a region outside the 3D outlinefrom the plurality of background-removed in-focus images. In this way,sectional images of the object each corresponding to one of theplurality of background-removed in-focus images are generated. Thesesectional images are sectional images at focal planes set in advance andare referred to as reference sectional images. The 3D image generationunit 250 generates a 3D model of the object by using the plurality ofreference sectional images. This 3D model can include informationregarding the 3D outline of the object and the plurality of referencesectional images.

The image operation unit 260 is implemented by, for example, a controlcircuit or a processor. The image operation unit 260 displays the 3Dmodel of the object generated by the 3D image generation unit 250 on thedisplay unit 150. At that time, the image operation unit 260 displaysthe 3D outline of the object or the like on the display unit 150.Further, in response to selection of a position of the section of theobject, the image operation unit 260 displays a brief image of theselected section on the display unit 150 together with the 3D outline ofthe object. At that time, the 3D image generation unit 250 generates apreview sectional image, which is the brief sectional image of theobject, by using information included in the plurality of referencesectional images. Accordingly, for example, when the selected section isa section that crosses the focal planes (hereinafter, referred to asreference sectional image planes) corresponding to the plurality ofreference sectional images, regions between the plurality of referencesectional image planes are not clearly shown in the preview sectionalimage. The image operation unit 260 accepts, via the CG operation inputunit 140, an instruction to select a position of the preview sectionalimage to be displayed and displays the preview sectional image of theobject on the basis of this instruction.

The sectional image generation unit 270 is implemented by, for example,a control circuit or a processor. The sectional image generation unit270 generates a detailed image of a to-be-displayed section of theobject by using the position of the to-be-displayed section of theobject and information included in the plurality of reference sectionalimages and displays the detailed sectional image on the display unit150. Specifically, the sectional image generation unit 270 uses, as apixel value of a pixel in a region of the to-be-displayed section of theobject that overlaps or crosses a reference sectional image plane, apixel value of the corresponding pixel of the reference sectional image.The sectional image generation unit 270 calculates a pixel value of apixel in a region of the to-be-displayed section of the object thatneither overlaps nor crosses any reference sectional image plane, byusing pixel values of pixels of reference sectional images at respectivereference sectional image planes that are located near this pixel. Thatis, the sectional image generation unit 270 calculates a pixel value ofa pixel (hereinafter, also referred to an interpolation pixel) locatedbetween pixels for which respective pixels of the correspondingreference sectional images are used in the to-be-displayed section ofthe object. The sectional image generation unit 270 then generates adetailed image of the section of the object by using the pixel values ofpixels included in the reference sectional images and the pixel valuesof interpolation pixels. The sectional image generation unit 270displays the generated image on the display unit 150. The sectionalimage generation unit 270 can be informed of the position of theto-be-displayed section of the object via the CG operation input unit140. Specifically, the to-be-displayed section of the object may beselected by using the CG operation input unit 140 from among previewsectional images of the object displayed on the display unit 150. Inthis way, the to-be-displayed section of the object may be determined.

1-2. Operation of Image Generation System

1-2-1. Overview of Operation of Refocusing Process of Image GenerationSystem

An overview of an operation of the refocusing process performed by theimage generation system 10 thus configured, that is, an overview of anoperation of generating an in-focus image, will be described next. FIG.7 is a flowchart illustrating an example of an operation of generatingan in-focus image performed by the image generation system 10 accordingto the first embodiment. FIG. 8 is a schematic diagram illustrating anexample of a relationship between coordinates and a focal plane.

In step S1100, the imaging control unit 103 of the imaging device 100irradiates the object with light by sequentially using the plurality ofilluminators 101 and captures a plurality of images of the object.Specifically, the imaging control unit 103 records intensity of lightthat has reached each pixel on the light-receiving surface of the imagesensor 102 when the object is irradiated with light by each of theplurality of illuminators 101. In this way, the imaging control unit 103captures images of the object. Each captured image is stored in thestorage unit 120 together with the position information of theilluminator 101 that has irradiated the object with light at the time ofimaging. In this embodiment, the positions of the plurality ofilluminators 101 are fixed with respect to the image sensor 102, andthus the position information of each of the plurality of illuminators101 is predetermined. Details of the imaging process will be describedlater.

In step S1200, the focal plane determination unit 210 of the imagegeneration device 200 determines the focal plane. Specifically, thefocal plane determination unit 210 determines the position and tilt(angle) of the focal plane with respect to the image sensor 102. Forexample, the focal plane determination unit 210 may determine the focalplane on the basis of information regarding predetermined focal planesstored in the first memory 121. Alternatively, the focal planedetermination unit 210 may determine the focal plane on the basis ofspecifying information that is accepted from the user via the focalplane input unit 130 and that specifies the focal plane. The focal planecorresponds to a virtual plane for which an in-focus image is generated.That is, a plurality of pixels included in an in-focus image of anobject at a focal plane and a plurality of points on the focal planehave a one-to-one correspondence. For example, the focal planedetermination unit 210 determines the focal plane by using the angle andposition of the focal plane. The angle and position of the focal planeare defined by using an xyz space illustrated in FIG. 8, for example.

Referring to FIG. 8, the x-y plane matches the light-receiving surfaceof the image sensor 102. The z axis is orthogonal to the light-receivingsurface of the image sensor 102. In this case, the angle of the focalplane is defined using angles with respect to the x axis and y axis inthe xyz space having the original at the center of the light-receivingsurface of the image sensor 102. The position of the focal plane isdefined by coordinates of the center of the focal plane.

In step S1300, the refocusing processing unit 220 performs a refocusingprocess on the basis of the plurality of captured images, the positioninformation of the plurality of illuminators 101, and informationregarding the focal plane and determines a pixel value of each pixel(i.e., each point) on the focal plane. Details of the refocusing processwill be described later.

In step S1400, the in-focus image generation unit 230 generates anin-focus image at the focal plane, which is image data that can beoutput to a display, on the basis of the result of the refocusingprocess performed in step S1300.

1-2-2. Imaging Process

Now, details of the operation performed by the imaging device 100 instep S1100, specifically, the operation of the imaging control unit 103,is described. FIG. 9 is a flowchart illustrating an example of theoperation of the imaging device 100.

In step S1110, the imaging control unit 103 determines whether imagingof an object irradiated with light from illumination positions has beenfinished with reference to a list (hereinafter, referred to as anillumination position list) of a plurality of predetermined illuminationpositions or a plurality of illumination positions specified through aninput from outside (not illustrated). The plurality of illuminators 101and the plurality of illumination positions included in the illuminationposition list have a one-to-one correspondence.

If imaging is finished for illumination from all the illuminationpositions included in the illumination position list (yes in stepS1110), the process proceeds to step S1200 to determine the focal plane.On the other hand, if imaging is not finished for illumination from anyof the illumination positions included in the illumination position list(no in step S1110), the process proceeds to step S1120.

In step S1120, the imaging control unit 103 selects an illuminationposition from which light has not been radiated yet from among theplurality of illumination positions included in the illuminationposition list and outputs control signals to the plurality ofilluminators 101. If light has not been radiated from a plurality ofillumination positions, the imaging control unit 103 selects oneillumination position from among the plurality of illuminationpositions. Each illumination position is represented, for example, by anumber assigned to the illumination position in the illuminationposition list. Alternatively, each illumination position is represented,for example, by coordinate values in the xyz space illustrated in FIG.8. The illumination position is selected in ascending order of the list,for example.

In step S1130, the plurality of illuminators 101 start irradiating theobject with light in accordance with the control signals output from theimaging control unit 103 in step S1120. That is, the illuminator 101located at the illumination position selected in step S1120 startsirradiating the object with light.

In step S1140, while the object is irradiated with light by theilluminator 101, the image sensor 102 captures an image formed by lightthat has been emitted from the illuminator 101 and has passed throughthe object.

In step S1150, the imaging control unit 103 then outputs control signalsto the plurality of illuminators 101 to stop irradiation of the objectwith light. Irradiation of the object with light need not be stopped inaccordance with the control signals output from the imaging control unit103. For example, the plurality of illuminators 101 may count a periodfrom the start of irradiation and may spontaneously stop irradiationupon the counted period exceeding a predetermined period. Alternatively,the image sensor 102 may output control signals for stopping irradiationto the plurality of illuminators 101 after the image sensor 102 finishescapturing an image in step S1140.

In step S1160, the imaging control unit 103 then outputs data of theimage captured in step S1140 and the position information of theilluminator 101 used in step S1130 to the storage unit 120. The storageunit 120 then stores the image data and the illumination positioninformation in association with each other. The process returns to stepS1110 after step S1160.

As a result of iteration of the process from step S1110 to step S1160,the object is sequentially irradiated with light by the illuminators 101located at the respective illumination positions included in theillumination position list and an image is captured every time theobject is irradiated with light.

1-2-3. Detailed Operation of Refocusing Process of Image GenerationSystem

A detailed operation performed by the refocusing processing unit 220 instep S1300 will be further described. FIG. 10 is a flowchartillustrating an example of an operation of the refocusing processingunit 220 according to the first embodiment. FIGS. 11 to 15 are schematicdiagrams for describing a specific example of a calculation methodperformed in the refocusing process.

Each step of FIG. 10 will be described below with reference to FIGS. 11to 15.

In step S1310, the refocusing processing unit 220 obtains informationregarding the focal plane determined in step S1200 from the focal planedetermination unit 210.

The information regarding the focal plane includes, for example,coordinate values of the center of the focal plane and a valuerepresenting the tilt of the focal plane. The tilt of the focal plane isrepresented by, for example, an angle between a line of intersection ofthe focal plane and the x-z plane and the x axis. In addition, the tiltof the focal plane is represented by, for example, an angle between aline of intersection of the focal plane and the y-z plane and the yaxis. The coordinate values of the center of the focal plane arecoordinate values of a point on the focal plane that corresponds to thepixel located at the center of the in-focus image.

FIG. 11 illustrates an example of a sectional view of the imaging device100 and an object 1000 at the x-z plane. The object 1000 is locatedbetween illuminators 101 a and 101 b and the image sensor 102 and islocated on the image sensor 102. The refocusing processing unit 220obtains information regarding a focal plane 1100.

In step S1320, the refocusing processing unit 220 determines whetherrefocusing processing has been finished for all pixels included in anin-focus image. Herein, the refocusing processing indicates processingfrom step S1320 to step S1390.

If the refocusing processing has been finished for all pixels includedin the in-focus image (yes in step S1320), the refocusing processingunit 220 ends the refocusing process (and the process proceeds to stepS1400).

If the refocusing processing has not been finished for any of the pixelsincluded in the in-focus image (no in step S1320), the refocusingprocessing unit 220 continues the refocusing process (and the processproceeds to step S1330).

The in-focus image includes a plurality of pixels. The plurality ofpixels included in the in-focus image and a plurality of points on thefocal plane have a one-to-one correspondence. FIG. 12 illustrates aplurality of points 1102 a to 1102 e on the focal plane 1100 thatcorrespond to a plurality of pixels included in the in-focus image. Theplurality of points 1102 a to 1102 e on the focal plane 1100 illustratedin FIG. 12 are points on the object 1000; however, points that are noton the object 1000 may correspond to respective pixels of the in-focusimage.

In step S1330, the refocusing processing unit 220 selects a pixel fromamong the plurality of pixels included in the in-focus image. The pixelselected in this step is a pixel for which the refocusing processing hasnot been performed yet from among the plurality of pixels included inthe in-focus image. Note that the initial pixel value of the in-focusimage is equal to zero.

For example, information regarding pixel(s) of the in-focus image forwhich the refocusing processing has already been performed is stored inthe second memory 122 illustrated in FIG. 1. After processing of stepS1390 (described later) is performed, the refocusing processing unit 220stores information regarding the pixel subjected to the refocusingprocessing in the second memory 122. The refocusing processing unit 220selects a pixel for which the refocusing processing has not beenperformed yet with reference to the information regarding the pixel(s)stored in the second memory 122. A case where a pixel corresponding tothe point 1102 a is selected as illustrated in FIG. 13 will be describedbelow. The pixel corresponding to the point 1102 a is also referred toas a selected pixel.

In step S1340, the refocusing processing unit 220 determines whetheraddition processing has been finished for all the illuminationpositions.

If addition processing has been finished for all the illuminationpositions (yes in step S1340), the process of the refocusing processingunit 220 returns to step S1320.

On the other hand, if addition processing has not been finished yet forany of the illumination positions (no in step S1340), the refocusingprocessing unit 220 continues the addition processing (and the processproceeds to step S1350). Here, addition processing indicates processingfrom step S1340 to step S1390.

In step S1350, the refocusing processing unit 220 selects anillumination position for which addition processing has not beenfinished yet from among all the illumination positions used for imaging.

In step S1360, the refocusing processing unit 220 calculates theposition of a point at which a line that passes through the selectedillumination position and the position corresponding to the selectedpixel on the focal plane crosses the light-receiving surface of theimage sensor 102.

FIG. 14 illustrates an intersection point 1103 a of a line 1200 thatpasses through the position of the illuminator 101 a and the point 1102a corresponding to the selected pixel and the light-receiving surface ofthe image sensor 102. Hereinafter, the intersection point 1103 a is alsoreferred to as a target point that is a point subjected to additionprocessing.

The target point on the light-receiving surface of the image sensor 102is represented by coordinate values on the x-y plane illustrated in FIG.8, for example.

In step S1370, the refocusing processing unit 220 obtains an imagecorresponding to the selected illumination position from the storageunit 120. That is, the refocusing processing unit 220 obtains an imagecaptured by using the illuminator 101 located at the selectedillumination position from the storage unit 120. Specifically, therefocusing processing unit 220 obtains an image stored in the storageunit 120 in accordance with a correspondence between the illuminationposition information and the image illustrated in FIG. 6. For example,the refocusing processing unit 220 obtains an image corresponding to theposition of the illuminator 101 a illustrated in FIG. 13.

In step S1380, the refocusing processing unit 220 determines a positionin the obtained image that corresponds to the position of the targetpoint on the image sensor 102 determined in step S1360. Specifically,the refocusing processing unit 220 determines a position in the obtainedimage that corresponds to the position of the target point withreference to a pixel array of the obtained image.

If the position in the photographic image that corresponds to the targetpoint is between the plurality of pixels, the refocusing processing unit220 calculates a pixel value of the target point in the photographicimage by performing interpolation processing by using pixel values ofthe plurality of pixels adjacent to the position of the target point.Specifically, the refocusing processing unit 220 determines a distancesbetween the target point and each of a plurality of pixels (for example,four pixels) that are adjacent to the target point and multiplies pixelvalues of the respective pixels by the respective proportions of thedistances between the target point and the pixels and adds the productstogether. In this way, the refocusing processing unit 220 determines thepixel value of the target point in the photographic image.

FIG. 15 is a schematic diagram for describing how the pixel value of thetarget point is calculated in step S1380. Referring to FIG. 15,distances between the target point and four pixels A to D that areadjacent to the target point are respectively denoted as a, b, c, and d.In this case, the pixel value L_(t) of the target point is determined byusing Equation 3 below.

$\begin{matrix}{L_{t} = {\left( {\frac{L_{a}}{a} + \frac{L_{b}}{b} + \frac{L_{c}}{c} + \frac{L_{d}}{d}} \right) \times \left( {a + b + c + d} \right)}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

In Equation 3, L_(a), L_(b), L_(c), and L_(d) represent pixel values ofthe pixels A, B, C, and D, respectively.

In step S1390, the refocusing processing unit 220 adds the pixel valueof the target point calculated in step 1380 to the pixel value of theselected pixel in the in-focus image.

As a result of iteration of the processing from step S1340 to stepS1390, the result obtained by adding the pixel value of the target pointin the respective images captured for all the illumination positions tothe pixel value of the selected pixel is calculated as the pixel valueof the selected pixel.

As a result of such addition processing, a plurality of images formed bylight that has passed through each point on the focal plane from aplurality of directions are superimposed at a pixel in the in-focusimage.

Referring to FIG. 14, light radiated from the illuminator 101 a passesthrough the point 1102 a on the focal plane 1100 that corresponds to theselected pixel and reaches the target point (intersection point 1103 a)on the light-receiving surface of the image sensor 102. Accordingly, animage at the point 1102 a on the focal plane 1100 is contained at aposition of the target point (i.e., the intersection point 1103 a) ofthe image captured using the illuminator 101 a.

In addition, in FIG. 14, light radiated from the illuminator 101 bpasses through the point 1102 a on the focal plane 1100 that correspondsto the selected pixel and reaches the target point (intersection point1103 b) on the light-receiving surface of the image sensor 102.Accordingly, an image at the point 1102 a on the focal plane 1100 iscontained at a position of the target point (i.e., the intersectionpoint 1103 b) in the image captured using the illuminator 101 b.

By adding the images (pixel values) at such target points (i.e., theintersection points 1103 a and 1103 b) together, a plurality of imagesformed by using light from a plurality of directions are superimposed atthe selected pixel in the in-focus image.

According to the refocusing process described above, the pixel value ofa target point, which is an intersection point of a line connecting theposition of a pixel on the focal plane and the position of theilluminator 101 and the light-receiving surface of the image sensor 102,can be used for the pixel value of the pixel. Accordingly, pixel valuesof a pixel in a plurality of captured images are successfully reflectedin a pixel value of the corresponding pixel of an in-focus image at avirtual focal plane, and consequently a high-quality in-focus image ofthe object is successfully generated.

A target point which is an intersection point of the light-receivingsurface of the image sensor 102 and a line that passes through theposition of the illuminator 101 and a point corresponding to theselected pixel is used in the refocusing process described above. Thatis, the target point having a pixel value to be added to the pixel valueof the selected pixel is identified on the basis of a relationship amongthe position of the illuminator 101, the point corresponding to theselected pixel, and the target point on the light-receiving surface ofthe image sensor 102; however, a pixel having the pixel value to beadded to the pixel value of the selected pixel may be identified on thebasis of a relationship among the position of the illuminator 102, theselected pixel, and the pixel on the light-receiving surface of theimage sensor 102 instead of the relationship between the points. Forexample, a pixel value detected by a pixel of the image sensor 102 thatreceives light that reaches the light-receiving surface of the imagesensor 102 after being emitted from the illuminator 101 and passingthrough a region of the selected pixel may be added to the pixel valueof the selected pixel. That is, a pixel value detected by a pixel of theimage sensor 102 that satisfies a positional relationship in which theposition of the illuminator 101, the selected pixel, and the pixel onthe image sensor 102 are arranged in a line may be added to the pixelvalue of the selected pixel.

1-2-4. Detailed Operation of Image Generation System

An operation of the image generation system 10 thus configured will bedescribed in detail with reference to FIG. 16 next. FIG. 16 is aflowchart illustrating an example of an operation of the imagegeneration system 10 according to the first embodiment. The followingdescription will be given of the case where a translucent embryo S whichis an early embryo of a vertebrate animal as illustrated in FIG. 17 isused as an object to be imaged. FIG. 17 is a perspective view of theembryo S, which is an example of the object. Referring to FIG. 17, thespherical embryo S includes two spherical cells S1 and S2 resulting fromcleavage.

In step S2100, the imaging control unit 103 of the imaging device 100irradiates the translucent embryo S placed on the light-receivingsurface of the image sensor 102 with light by sequentially using theplurality of illuminators 101 and captures a plurality of images of theembryo S. In the first embodiment, the positions of the plurality ofilluminators 101 are fixed with respect to the image sensor 102, andposition information of the plurality of illuminators 101 ispredetermined. The imaging control unit 103 records intensity of lightthat has reached individual pixels on the light-receiving surface of theimage sensor 102 when the embryo S is irradiated with light by each ofthe plurality of illuminators 101 so as to capture an image of theembryo S. Each captured image is stored in the storage unit 120 togetherwith the position information of the illuminator 101 used forirradiation of the embryo S with light at the time of imaging.

In step S2200, the focal plane determination unit 210 of the imagegeneration device 200 determines a plurality of reference focal planeswhich are focal planes used as references. In the first embodiment, theplurality of reference focal planes that are substantially parallel tothe light-receiving surface of the image sensor 102 and are spaced apartfrom one another are determined. Specifically, a plurality of referencefocal planes that are substantially parallel to the light-receivingsurface and are spaced apart from one another by an interval ofapproximately 1 μm are determined for the embryo S having a diameter ofapproximately 100 μm. The plurality of reference focal planes arearranged in a region that extends over a distance of approximately 110μm from the light-receiving surface. With this configuration, the entireembryo S is included in a region where the plurality of reference focalplanes are present. Many reference focal planes among the plurality ofreference focal planes cross the embryo S. Note that the specificnumerical values are example values and numerical values of therespective elements are not limited to these numerical values.

In step S2300, the refocusing processing unit 220 performs therefocusing process on the basis of the plurality of captured images, theposition information of the plurality of illuminators 101, and theinformation regarding the reference focal planes and determines pixelvalues of respective pixels (i.e., points) on each of the referencefocal planes.

In step S2400, the in-focus image generation unit 230 generatesreference in-focus images, which are in-focus images at the plurality ofreference focal planes and can be output to a display or the like, onthe basis of the result of the refocusing process performed in stepS2300. Note that since the reference focal plane is a plane at which anin-focus image is formed, it is also referred to as an in-focus imageplane or a refocusing image plane. FIG. 18 illustrates referencein-focus images Ia at a plurality of reference focal planes FP of theembryo S. FIG. 18 is a schematic diagram illustrating a stack of thereference in-focus images Ia at the plurality of reference focal planesFP arranged in a line in the arrangement order. FIG. 18 illustrates someof the plurality of reference in-focus images Ia. FIG. 19 illustrates anexample in which the reference in-focus images Ia arranged in a line aredisplayed as an image on the display screen 151 of the display unit 150.FIG. 19 is a diagram illustrating an example in which the referencein-focus images Ia are displayed using a photograph. FIG. 19 illustratesan example in which the embryo S includes four cells. The referencein-focus image Ia at each reference focal plane FP includes a region ofpixels that receives light that has passed through the embryo S, thatis, an object-showing region M including an image of the embryo S.

In step S2500, the target object extraction unit 240 identifies anoutline of the object-showing region M including the image of the embryoS in each reference in-focus image Ia at the corresponding referencefocal plane FP as illustrated in FIG. 20 and generates abackground-removed in-focus image Ib by removing background locatedoutside the outline from the reference in-focus image Ia. The outline ofthe embryo S may be identified in the following manner, for example. Theshape of the embryo S is substantially spherical, and the radius of theembryo S is approximately 100 μm. The target object extraction unit 240applies a Laplacian filter to the reference in-focus image Ia to extractedge points. The target object extraction unit 240 classifies thefiltering results based on a threshold. For example, the target objectextraction unit 240 sets a threshold on the basis of an intensitydistribution of the input image (the reference in-focus image Ia in thiscase). For example, the threshold is a point at which the ratio of thenumber of low-intensity pixels to the number of all pixels is 25% basedon the intensity histogram. The target object extraction unit 240extracts, as edge points, pixels having an intensity lower than or equalto the threshold. The target object extraction unit 240 applies theHough transform on the extracted edge points to extract circles. Forexample, the target object extraction unit 240 extracts a circlecorresponding to the embryo S by extracting circles having a radius in arange from 80 μm to 120 μm. Since the center and the radius of thecircle are determined through the Hough transform, a circle serving asan outline is successfully identified in the reference in-focus imageIa. FIG. 20 is a schematic diagram illustrating a stack of the pluralityof background-removed in-focus images Ib arranged in a line inaccordance with the arrangement order of the reference focal planes FPas in FIG. 18. FIG. 21 illustrates an example in which thebackground-removed in-focus images Ib that are stacked in a line aredisplayed on the display screen 151 of the display unit 150 as an image.FIG. 21 is a diagram illustrating an example in which thebackground-removed in-focus images Ib are displayed using a photograph.FIG. 21 illustrates the example in which the embryo S includes fourcells.

In step S2600, the 3D image generation unit 250 extracts an outlinereference in-focus image Iba in which the object-showing region M of theembryo S has an outline having the highest contrast from among theplurality of background-removed in-focus images Ib as illustrated inFIG. 22. The 3D image generation unit 250 may determine which of a firstoutline reference in-focus image Iba1 and a second outline referencein-focus image Iba2 includes an outline having a higher contrast in thefollowing manner. For example, the first outline reference in-focusimage Iba1 includes a1 pixels determined to be an outline. The 3D imagegeneration unit 250 determines, for each of the pixels determined to bean outline, the largest value from among the pixel value differencesbetween the pixel and the plurality of adjacent pixels. Let A1 denotethe sum of the largest values. For example, the second outline referencein-focus image 1 ba 2 includes a2 pixels determined to be an outline.The 3D image generation unit 250 determines, for each of the pixelsdetermined to be an outline, the largest value from among the pixelvalue differences between the pixel and the plurality of adjacentpixels. Let A2 denote the sum of the largest values. The 3D imagegeneration unit 250 determines that the first outline reference in-focusimage Iba1 includes an outline having a higher contrast than the secondoutline reference in-focus image Iba2 if {(A1)/(a1)}>{(A2)/(a2)}. FIG.22 is a schematic diagram illustrating an example of the outlinereference in-focus image Iba. The pixel value of a pixel that makes thecontrast of the outline of the object-showing region M the highestcorresponds to a pixel value obtained when the pixel of the image sensor102 detects light that passes through the periphery of the embryo S anda region near the periphery when the embryo S is irradiated with lightfrom directly above the embryo S in a direction perpendicular to thelight-receiving surface of the image sensor 102. Accordingly, a circularoutline Ma of the object-showing region M in the outline referencein-focus image Iba corresponds to an outline of a section that passesthrough the center of the embryo S, that is, a two-dimensional outlinefor the embryonic membrane of the embryo S, and corresponds to a planeroutline of the embryo S when the embryo S is viewed in a directionperpendicular to the light-receiving surface of the image sensor 102.The 3D image generation unit 250 then extracts the outline Ma of theembryo S from the outline reference in-focus image Iba. In this way, thetwo-dimensional outline Ma of the embryo S is extracted. Which of thefirst outline reference in-focus image Iba1 and the second outlinereference in-focus image Iba2 includes an outline having a highercontrast may be determined in the following manner. The first outlinereference in-focus image Iba1 includes a1 pixels determined to be anoutline, and A1 denotes the sum of pixel values of the a1 pixels. Thesecond outline reference in-focus image Iba2 includes a2 pixelsdetermined to be an outline, and A2 denotes the sum of pixel values ofthe a2 pixels. In this case, the 3D image generation unit 250 determinesthat the first outline reference in-focus image Iba1 includes an outlinehaving a higher contrast than the second outline reference in-focusimage Iba2 if {(A1)/(a1)}>{(A2)/(a2)}.

In step S2700, the 3D image generation unit 250 identifies athree-dimensional outline of the embryo S from the shape of thetwo-dimensional outline extracted from the outline reference in-focusimage Iba and the shape of the embryo S stored in the first memory 121.For example, the first memory 121 stores information indicating that“the embryo S has a spherical shape”. Specifically, the 3D imagegeneration unit 250 identifies that the three-dimensional outline Sc ofthe embryo S is a sphere having a radius equal to the radius of theoutline Ma from the circular outline Ma of the object-showing region Mof the outline reference in-focus image Iba and the information storedin the first memory 121 indicating that the embryo S has a sphericalshape as illustrated in FIGS. 22 and 23. FIG. 23 is a schematic diagramillustrating an example of the three-dimensional outline Sc of theembryo S.

In step S2800, the 3D image generation unit 250 generates referencesectional images Ic by associating the plurality of background-removedin-focus images Ib with the three-dimensional outline Sc of the embryo Sand removing a region outside the three-dimensional outline Sc from theplurality of background-removed in-focus images Ib. The referencesectional images Ic correspond to sectional images of the embryo S atthe plurality of reference focal planes FP determined in step S2200.

Specifically, the 3D image generation unit 250 identifies the positionalrelationship between the reference focal plane FP corresponding to eachbackground-removed in-focus image Ib and the spherical outline Sc of theembryo S. At that time, the 3D image generation unit 250 determines thatthe reference focal plane FP including the outline reference in-focusimage Iba is a plane that passes through the center of thethree-dimensional outline Sc and that is substantially parallel to thelight-receiving surface of the image sensor 102. The 3D image generationunit 250 further determines the position of the reference focal plane FPincluding another background-removed in-focus image Ib with respect tothe three-dimensional outline Sc on the basis of the abovedetermination. The 3D image generation unit 250 then removes a regionoutside the spherical outline Sc of the embryo S from the plurality ofbackground-removed in-focus images Ib and generates the plurality ofreference sectional images Ic as illustrated in FIG. 24. FIG. 24 is aschematic diagram illustrating an example of the plurality of referencesectional images Ic as in FIG. 18. FIG. 24 illustrates the plurality ofreference sectional images Ic that are arranged in a line in thearrangement order of the reference focal planes FP. Each referencesectional image Ic is a sectional image of the embryo S at thecorresponding reference focal plane FP. In addition, FIG. 25 illustratesan example in which the plurality of reference sectional images Icobtained by removing a region outside the spherical outline Sc of theembryo S are displayed as an image on the display screen 151 of thedisplay unit 150. FIG. 25 is a diagram illustrating an example in whichthe reference sectional images Ic are displayed using a photograph. Allthe reference sectional images Ic are illustrated in the state A of FIG.25 and form the outer shape of the spherical outline Sc of the embryo S,whereas some of the reference sectional images Ic are displayed in thestate B of FIG. 25.

The background-removed in-focus images Ib need not be generated in stepS2500. For example, background outside the outline of the object-showingregion M may be removed from the reference in-focus images Ia throughprocessing for removing a region outside the spherical outline Sc of theembryo S in step S2800. That is, the processing of step S2500 may beincluded in the processing of step S2800. in this case, the processingof step S2600 to step S2800 is performed on the reference in-focusimages Ia.

In step S2900, the 3D image generation unit 250 generates a 3D model A,which is a 3D model of the embryo S, as image data that can be output toa display or the like by using the three-dimensional outline Sc of theembryo S and the plurality of reference sectional images Ic. The 3Dmodel A includes information regarding the spherical outline Sc of theembryo S and the plurality of reference sectional images Ic and includesthe reference sectional images Ic as sectional images. The 3D model A issimilar to that of the state A illustrated in FIG. 25 when it isdisplayed as an image on the display screen 151 of the display unit 150.

In step S3000, the image operation unit 260 displays the 3D model A ofthe embryo S generated by the 3D image generation unit 250 on thedisplay unit 150 in an operable manner. The image operation unit 260moves the 3D model A of the embryo S on the display unit 150 inaccordance with an instruction input via the CG operation input unit140. The image operation unit 260 accepts selection of a section atvarious positions of the embryo S via the CG operation input unit 140.Further, the image operation unit 260 may display a preview sectionalimage Bi, which is a brief image of the selected section, on the 3Dmodel A of the embryo S or separately from the 3D model.

For example, the image operation unit 260 displays the 3D model A of theembryo Son the display screen 151 of the display unit 150 in an operablestate as illustrated in FIGS. 26A, 26B, and 26C. Each of FIGS. 26A, 26B,and 26C is a diagram illustrating an example of the 3D model of theembryo S displayed on the display screen 151 of the display unit 150.The screen illustrated in FIGS. 26A, 26B, and 26C is a screen in whichthe user is allowed to select and display a section to be displayed indetail by using the 3D model A of the embryo S.

In the display screen 151, the position of the 3D model A of the embryoS is defined by using the xyz space. The x-y plane is set to be parallelto the light-receiving surface of the image sensor 102. Further, areference axis C that passes through the center of the 3D model A of thespherical embryo S is set, and the orientation of the reference axis Cis defined by using the xyz space. In addition, the position of asection B of the 3D model A of the embryo S that is perpendicular to thereference axis C is defined by using the xyz space.

As illustrated in FIGS. 26A and 26B, the reference axis C rotates withrespect to the spherical center Ac of the 3D model A in the displayscreen 151 in accordance with an instruction input via the CG operationinput unit 140, and consequently the orientation of the reference axis Cis changed in any manner. A horizontal movement scroll bar 152 and avertical movement scroll bar 153 are displayed near the edges of thedisplay screen 151. The reference axis C rotates in a direction alongthe x-y plane with respect to the spherical center Ac of the 3D model A,that is, rotates around an axis parallel to the z axis that passesthrough the center of the 3D model A, in response to a scroll operationon the horizontal movement scroll bar 152. The reference axis C rotatesin a direction along the y-z plane with respect to the center of the 3Dmodel A, that is, rotates around an axis parallel to the x axis thatpasses through the center of the 3D model A in response to a scrolloperation on the vertical movement scroll bar 153. In this way, theorientation of the reference axis C is changed in any given manner. Inaddition, the direction vector of the reference axis C is displayed byusing the x, y, and z coordinates at a movement position display portion155 located at an upper right portion of the display screen 151. Theorientation of the reference axis C may be determined by inputtingnumerical values at fields of the x, y, and z coordinates of themovement position display portion 155. Note that the configuration usedto move the reference axis C is not limited to the above one and anyother configuration may be used.

Further, the section B of the 3D model A slides along the reference axisC in the direction of the axis in any given manner in accordance with aninstruction input via the CG operation input unit 140 as illustrated inFIGS. 26B and 26C. The preview sectional image Bi, which is a brieflycreated image of the section B, is displayed as the section B of the 3Dmodel A with the displayed content being changed in accordance with theposition of the section B. For example, FIG. 27 illustrates an examplein which the 3D model A illustrated in FIG. 26C is displayed as an imageon the display screen 151 of the display unit 150. FIG. 27 is a diagramillustrating an example in which part of the display screen 151illustrated in FIG. 26C is displayed using a photograph. Note that thepreview sectional image Bi may be displayed at a position different fromthat of the section B.

A section movement scroll bar 154 is displayed near an edge of thedisplay screen 151. The section B including the preview sectional imageBi slides (translates) along the reference axis C with the displayedcontent being changed in response to a scroll operation on the sectionmovement scroll bar 154. In addition, the position of the center Bc ofthe section B, which is the intersection point of the section B of the3D model A and the reference axis C, is displayed by using the x, y, andz coordinates at a section position display portion 156 at a middleright portion of the display screen 151. The position of the section Bmay be determined by inputting numerical values in fields of the x, y,and z coordinates of the section position display portion 156. Note thatthe configuration for moving the section B is not limited to the aboveone, and any other configuration may be used.

The preview sectional image Bi at a given position of the 3D model A canbe displayed on the display screen 151 of the display unit 150 byrotating the reference axis C and sliding the section B in combination.In this example, the reference axis C and the section B move but the 3Dmodel A does not move; however, the 3D model A may be displayed so as tomove such as to rotate. Any given configuration may be used as theconfiguration for changing the position of the section B.

The preview sectional image Bi is generated by using informationincluded in the 3D model A, that is, information included in theplurality of reference sectional images Ic within the spherical outlineSc of the embryo S. For example, each pixel in a portion where thepreview sectional image Bi crosses the reference focal plane FP of thereference sectional image Ic is displayed in the preview sectional imageBi of the section B by using a pixel value of the corresponding pixel ofthe reference sectional image Ic. Each pixel in a region other than theintersection region is displayed by using a pixel value of thecorresponding pixel of the reference sectional image Ic located near thepixel or is not displayed. Accordingly, the preview sectional image Bishows a brief image of the section B.

Upon being supplied with position information of the section of theembryo S to be displayed in detail, the image generation system 10according to the first embodiment generates a detailed image of thesection and displays the detailed image on the display unit 150. At thattime, the sectional image generation unit 270 of the image generationdevice 200 generates a detailed image of the section and displays thedetailed image on the display unit 150. The position information of thesection of the embryo S to be displayed in detail may be supplied as aresult of the preview sectional image Bi of the embryo S currentlydisplayed on the display screen 151 of the display unit 150 beingselected and determined via the CG operation input unit 140.

An example of an operation performed by the image generation system 10to display a detailed sectional image of the embryo S will be describedwith reference to FIG. 28. FIG. 28 is a flowchart illustrating anexample of an operation performed by the image generation system 10according to the first embodiment to display a detailed sectional imageof the embryo S. The following description will be given of an examplein which the position of the section of the embryo S to be displayed indetail is determined by using the 3D model A of the embryo S displayedon the display unit 150.

In step S4100, the image operation unit 260 of the image generationdevice 200 displays the 3D model A of the embryo S so that operations,such as rotation of the reference axis C of the 3D model A and slidingof the section B, can be performed.

In step S4200, the image operation unit 260 rotates the reference axis Cof the 3D model A of the embryo S or slides the section B on the displayscreen 151 of the display unit 150 in accordance with an instructioninput via the CG operation input unit 150 as illustrated in FIGS. 26A to26C and displays various preview sectional images Bi of the 3D model A.Then, if the position of the displayed section B is at a desiredposition or the displayed preview sectional image Bi is a desiredsection, that is, if the desired section or the position of the desiredsection is displayed on the display screen 151, the user inputs aninstruction to display a detailed section via the CG operation inputunit 140 (yes in step S4200). Consequently, the position of the sectionof the embryo S to be displayed in detail is determined. At that time, arefocus icon 157 of the display screen 151 is selected through input viathe CG operation input unit 140 and the associated instruction isexecuted. As a result, the process proceeds to step S4300 so that thesectional image generation unit 270 generates the sectional image of theembryo S to be displayed. On the other hand, if an instructionassociated with the refocus icon 157 is not executed, that is, if aninstruction to display the section of the embryo S in detail is notissued (no in step S4200), the image operation unit 260 performs theoperation of step S4100.

In step S4300, the sectional image generation unit 270 generates adetailed image of the determined section by using the position of thedetermined section and information included in the plurality ofreference sectional images Ic. Specifically, the sectional imagegeneration unit 270 generates an in-focus image by using the determinedsection as the focal plane through the refocusing process. That is, thesectional image generation unit 270 uses a pixel value of acorresponding pixel of the reference sectional image Ic as a pixel valueof each pixel located in a region where the determined section overlapsor crosses the reference focal plane FP of the reference sectional imageIc. The sectional image generation unit 270 calculates a pixel value ofeach interpolation pixel, which is a pixel located in a region where thedetermined section neither overlaps nor crosses the reference focalplane FP of the reference sectional image Ic, by using pixel values ofpixels of the reference sectional images Ic at the reference focalplanes FP that are located near the interpolation pixel. The sectionalimage generation unit 270 then generates a detailed sectional image,which is an in-focus image of the determined section of the embryo S, asimage data that can be output to a display or the like by using thepixel values of the pixels of the reference sectional images Ic and thepixel values of the interpolation pixels and displays the detailedsectional image on the display screen 151.

As described above, the user of the image generation system 10 isallowed to display a desired preview sectional image Bi or the positionof the desired preview sectional image Bi of the 3D model A of theembryo S on the display screen 151 of the display unit 151 by operatingthe CG operation input unit 140. Further, the user is allowed to obtaina detailed sectional image of the preview sectional image Bi byoperating the CG operation input unit 140 and selecting the displayedpreview sectional image Bi or the position of the displayed previewsectional image Bi.

Now, a method for calculating a pixel value of a pixel of a section Bdto be displayed in detail in the case where the section Bd crosses thereference focal planes FP of the reference sectional images Ic will bedescribed with reference to FIGS. 29 and 30. FIG. 29 is a diagramillustrating an example case where pixel values of pixels of the sectionBd are estimated by using pixels of reference sectional images Ica andIcb respectively at two reference focal planes FPa and FPb that crossthe section Bd to be displayed in detail. FIG. 30 is a diagramillustrating a relationship between a pixel of the section Bd to bedisplayed in detail and pixels of the two reference sectional images Icaand Icb that are used to calculate the pixel of the section Bd.

The section Bd crosses the reference focal planes FPa and FPb of the tworeference sectional images Ica and Icb that are adjacent to each other.A pixel Bda of the section Bd for which a pixel value is to be estimatedis an interpolation pixel located between the reference focal planes FPaand FPb. The reference focal planes FPa and FPb are in-focus imageplanes located near the position of the interpolation pixel Bda. Thepixel value of the interpolation pixel Bda is estimated by using twopixels Ica1 and Ica2 of the reference sectional image Ica located nearthe position of the interpolation pixel Bda and two pixels Icb1 and Icb2of the reference sectional image Icb located near the position of theinterpolation pixel Bda. Accordingly, the reference focal planes FPa andFPb whose reference sectional images include pixels that influence thepixel value of the interpolation pixel Bda may be the ones that are theclosest to the position of the interpolation pixel Bda.

As illustrated in FIG. 30, distances from the position of theinterpolation pixel Bda to the pixels Ica1, Ica2, Icb1, and Icb2 aredenoted by a₁, a₂, b₁, and b₂, respectively. Pixel values of the pixelsIca1, Ica2, Icb1, and Icb2 are denoted by L_(a1), L_(a2), L_(b1), andL_(b2). Accordingly, the pixel value L_(Bda) of the interpolation pixelBda is determined by using Equation 4 below.

$\begin{matrix}{L_{Bda} = {\left( {\frac{L_{a_{1}}}{a_{1}} + \frac{L_{a_{2}}}{a_{2}} + \frac{L_{b_{1}}}{b_{1}} + \frac{L_{b_{2}}}{b_{2}}} \right) \times \left( {a_{1} + a_{2} + b_{1} + b_{2}} \right)}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Accordingly, the to-be-estimated pixel value L_(Bda) of theinterpolation pixel Bda of the section Bd is estimated by using pixelvalues of the pixels of the reference sectional images Ica and Icb atthe two reference focal planes FPa and FPb that are adjacent to thepixel Bda.

The in-focus image of the section Bd to be displayed in detail isgenerated by using pixel values of pixels of the reference sectionalimages Ic located at a portion where the section Bd overlaps or crossesthe respective reference focal planes FP of the reference sectionalimages Ic and pixel values of interpolation pixels that are calculatedby using pixel values of pixels of the reference sectional images Ic ofthe reference focal planes FP that are adjacent to the interpolationpixels. Accordingly, the in-focus image of the section B is generatedthrough a refocusing process in which information included in thereference sectional images Ic that are generated in advance is used. Forthis reason, the in-focus image of the section Bd is generated moreeasily and rapidly than in the case where the in-focus image isgenerated through a refocusing process using the plurality of capturedimages as in generation of the reference sectional images Ic. Note thatthree or more pixels of the reference sectional image Ica or three ormore pixels of the reference sectional image Icb may be used tocalculate the pixel value of the interpolation pixel Bda. In addition,the pixel value of each pixel on the section Bd is also successfullycalculated in a manner similar to the above-described one when thesection Bd to be displayed in detail is parallel to the reference focalplane FP of the reference sectional image Ic.

1-3. Advantageous Effects

As described above, the image generation device 200 according to thefirst embodiment is capable of generating a plurality of referencein-focus images corresponding to a plurality of reference focal planesthat pass through the embryo S serving as an object and of generating athree-dimensional image of the embryo S by using the generated referencein-focus images. The use of the reference in-focus images correspondingto the plurality of reference focal planes that pass through the embryoS may allow a three-dimensional image of the embryo S to be displayedthree-dimensionally including constituents, such as cells, included inthe embryo S even if the embryo S is translucent or transparent. Inaddition, since the in-focus images are generated for the plurality ofreference focal planes instead of generating the in-focus images for theentire region of the embryo S, an amount of processing needed togenerate the three-dimensional image of the embryo S can also bereduced.

In addition, the image generation device 200 according to the firstembodiment allows a given section to be selected by using thethree-dimensional image of the embryo S and allows an image of theselected section to be displayed. Since pixel values of a plurality ofsection pixels of a sectional image of the embryo S are calculated usingpixel values of the corresponding pixels of the reference in-focus imagelocated at the section pixels or pixel values of pixels of the referencein-focus images located near the section pixels, the sectional image ofthe embryo S can be a sharp image in which discontinuity and blur arereduced. Further, since pixel values of the section pixels of thesectional image of the embryo S are calculated by using pixels of thereference in-focus images of reference focal planes located on therespective sides of the section pixels, the pixel values of the sectionpixels can be highly accurate.

In addition, the image generation device 200 according to the firstembodiment generates a preview sectional image representing a section ofthe embryo S for preview and displays the preview sectional image on thedisplay screen. Pixel values of a plurality of pixels of the previewsectional image can be pixel values of respective pixels of thereference in-focus images located at the pixels of the preview sectionalimage. Thus, the user is allowed to select a to-be-displayed section ofthe embryo S with reference to the preview sectional image. In addition,since pixel values of pixels of the preview sectional image are pixelvalues of the respective pixels of the reference in-focus images, thepreview sectional image is generated easily.

In addition, the image generation device 200 according to the firstembodiment calculates a pixel value of each in-focus pixel by using, asthe pixel value of the in-focus pixel, a pixel value obtained by eachsensor pixel that satisfies a relationship in which the position of theilluminator 101, the position of the in-focus pixel in the referencein-focus image, and the position of the sensor pixel of the image sensor102 are on a line. Accordingly, pixel values of a plurality of capturedimages corresponding to each pixel of the in-focus image at the focalplane are successfully reflected in the pixel, and a high-qualityin-focus image of the embryo S is successfully generated.

Second Embodiment

A second embodiment will be described next. In the second embodiment,the image generation system displays an optimum sectional image, whichis a representative sectional image of the embryo S, on the displayscreen 151 of the display unit 150 prior to the 3D model A of the embryoS, which differs from the first embodiment. The second embodiment willbe described below in terms of differences from the first embodiment.

2-1. Configuration of Image Generation System

FIG. 31 is a block diagram illustrating an example of the functionalconfiguration of an image generation system 10A according to the secondembodiment. In the second embodiment, the plurality of illuminators 101of the imaging device 100 are arranged in a grid pattern on a flatsurface. An image generation device 200A further includes an optimumsection determination unit 280 and an optimum sectional image generationunit 290, compared with the image generation device 200 of the imagegeneration system 10 according to the first embodiment. The first memory121 stores early embryo models of the embryo S in advance in addition toinformation regarding predetermined focal planes and the shape of theembryo S, which is an object to be imaged. The image generation system10A includes an embryo model input unit 160 that accepts input of aninstruction to select a model of the embryo S to be displayed on thedisplay unit 150 from among the early embryo models stored in the firstmemory 121.

The early embryo models of the embryo S stored in the first memory 121include information regarding the number of cells included in the embryoS. For each early embryo model of the embryo S, the number of cellsincluded in the embryo S and a culture period from the start ofculturing of the embryo S may be stored in association with each other.In this case, the early embryo models of the embryo S may be stored as atable illustrated in FIG. 32. FIG. 32 illustrates an example of theearly embryo models stored in the first memory 121. In the exampleillustrated in FIG. 32, the model ID of each early embryo model, theculture period which indicates a period from the start of culturing ofthe embryo S, the number of cells included in the embryo S in theculture period, and an illustrative schematic diagram of the embryo S inthe culture period are stored in combination with one another.

The optimum section determination unit 280 is implemented by, forexample, a control circuit or a processor. The optimum sectiondetermination unit 280 determines the optimum section of the embryo S tobe initially displayed as an example when the embryo S is displayed onthe display unit 150. At that time, the optimum section determinationunit 280 determines the number of candidate optimum sections and thenumber of cells that can be included in the optimum section on the basisof the number of cells included in an embryo model selected via theembryo model input unit 160. The optimum section determination unit 280calculates the position of the optimum section on the basis of thedetermined elements and sends information representing the position ofthe optimum section to the refocusing processing unit 220.

Specifically, the optimum section determination unit 280 refers to anoptimum section setting table, such as the one illustrated in FIG. 33,stored in the first memory 121 in advance, and determines the elementsregarding the optimum section on the basis of this optimum sectionsetting table. After determining the elements, the optimum sectiondetermination unit 280 determines the position of the optimum section ofthe embryo S in a manner described later. FIG. 33 illustrates an exampleof the optimum section setting table stored in the first memory 121. Inthe example illustrated in FIG. 33, the number of cells included in theembryo S, the number of candidate optimum sections (i.e., the number ofcandidate optimum focal planes), and the largest number of cellspossibly included in an optimum sectional image which is a sectionalimage of the embryo S at the optimum focal plane are stored incombination with one another.

In the optimum section setting table illustrated in FIG. 33, if there isonly one candidate optimum focal plane and the optimum focal planepasses through the center of a cell included in the embryo S or a regionof 5 μm from the center, the largest number of cells which the optimumfocal plane passes through is used as the largest number of cellspossibly included in the optimum sectional image at the optimum focalplane. Alternatively, if the optimum focal plane passes through aposition at which distances from the respective centers of individualcells become the smallest, the largest number of cells which the optimumfocal plane can pass through is used as the largest number of cellspossibly included in the optimum sectional image at the optimum focalplane. At that time, a situation where distances from the respectivecenters of individual cells become the smallest may indicate a situationwhere the sum of the distances from the respective centers of theindividual cells becomes the smallest.

If there are two or more candidate optimum focal planes, the number ofoptimum focal planes is set in accordance with the number of cellsincluded in the embryo S. Further, if the optimum focal plane passesthrough a position at which distances from the respective centers ofindividual cells become the smallest, the largest number of cells whichthe optimum focal plane can pass through is used as the largest numberof cells possibly included in the optimum sectional image at the optimumfocal plane. At that time, a situation where distances from therespective centers of a plurality of cells become the smallest mayindicate a situation where the sum of the distances from the respectivecenters of the plurality of cells becomes the smallest or the sum of thedistances from the respective centers of all the cells becomes thesmallest.

The optimum sectional image generation unit 290 is implemented by, forexample, a control circuit or a processor. The optimum sectional imagegeneration unit 290 generates an in-focus image at the optimum focalplane, that is, an optimum sectional image, from pixel values ofrespective pixels calculated by the refocusing processing unit 220 anddisplays the optimum sectional image on the display unit 150.

2-2. Operation of Image Generation System

An operation performed by the image generation system 10A thusconfigured to display an optimum sectional image on the display unit 150will be described. FIG. 34 is a flowchart illustrating an example of theoperation performed by the image generation system 10A according to thesecond embodiment to display an optimum sectional image. The followingdescription will be given of the case where the embryo S includes twocells S1 and S2 as illustrated in FIG. 17.

In step S5100, the optimum section determination unit 280 of the imagegeneration device 200A of the image generation system 10A displays onthe display unit 150 embryo models of the embryo S stored in the firstmemory 121.

In step S5200, the user of the image generation system 10A refers to atable of the embryo models of the embryo S, such as the one illustratedin FIG. 32, that is displayed on the display unit 150 and selects amodel ID of the embryo S for which the optimum sectional image is to bedisplayed on the display unit 150 via input using the embryo model inputunit 160. At that time, the user may select the embryo modelcorresponding to the time of imaging by comparing the culture period ofthe embryo S at the time of imaging using the illuminators 101 with theculture period included in the table of the embryo models.

In step S5300, the optimum section determination unit 280 determineselements of the optimum section, that is, the optimum focal plane, onthe basis of the number of cells included in the selected embryo modeland the optimum section setting table, such as the one illustrated inFIG. 33, that is stored in the first memory 121. Specifically, theoptimum section determination unit 280 determines the number of optimumfocal planes and the largest number of cells possibly included in asectional image at the optimum focal plane.

In step S5400, the optimum section determination unit 280 thendetermines the position of the optimum focal plane in the embryo S. Atthat time, the optimum section determination unit 280 obtains, from thestorage unit 120, a plurality of captured images each associated withposition information of a corresponding one of the plurality ofilluminators 101. The optimum section determination unit 280 furtherselects an illuminator group G including illuminators 101 arranged in aline from among the plurality of illuminators 101 arranged in a gridpattern on a flat surface as illustrated in FIG. 35. The optimum sectiondetermination unit 280 selects captured images captured when theilluminators 101 belonging to the illuminator group G have radiatedlight from among the obtained captured images and forms a captured imagegroup including the selected captured images. FIG. 35 is a diagramillustrating how the plurality of illuminators 101 are arranged when theplurality of illuminators 101 are viewed in a direction from theilluminators 101 toward the image sensor 102. The illuminators 101denoted by circles in FIG. 35 are point light sources and areimplemented by, for example, a light source and a pinhole having adiameter of 10 μm. Each circle in FIG. 35 indicates the center positionof the illuminator 101. The embryo S is placed substantially directlybelow the illuminator 101 denoted by a black circle in FIG. 35. Theembryo S has a diameter of approximately 100 μm. Further, the optimumsection determination unit 280 extracts lines that match the shape ofthe embryo S and the shapes of the two cells S1 and S2 included in theembryo S in each of the captured images belonging to the captured imagegroup. Specifically, since all of the embryo S and the two cells S1 andS2 have a spherical shape, the optimum section determination unit 280extracts, from each of the captured images, a circle representing theembryo S and circles representing the two cells S1 and S2 included inthe circle representing the embryo S. The outline of the embryo S may beidentified in the following manner, for example. The shape of the embryoS is substantially spherical, and the diameter of the embryo S isapproximately 100 μm. To extract edge points, a Laplacian filter isapplied to the reference in-focus image Ia. The filtering results areclassified based on a threshold. The threshold is set, for example,based on an intensity distribution of the input image (the referencein-focus image Ia in this case). For example, the threshold is set to avalue at which the ratio of the number of low-intensity pixels to thenumber of all pixels is equal to 25% based on the intensity histogram.Pixels having pixel values that are smaller than or equal to thethreshold are extracted as edge points. Hough transform is performed onthe extracted edge points to extract circles. At that time, a circle isextracted by setting the diameter to a value in a range from 80 μm to120 μm, for example. Consequently, a circle corresponding to the embryoS is extracted. Since the center and the radius of the circle arederived through Hough transform, a circle representing the outline issuccessfully identified in the reference in-focus image Ia. The cells S1and S2 are extracted by performing Hough transform by using edge pointslocated in the circle representing the determined outline of the embryoS. During Hough transform, circles are extracted by setting the diameterto a value in a range from 30 μm to 60 μm. Two circles are extractedwith overlapping permitted.

In step S5500, the optimum section determination unit 280 identifiesidentical elements included in the plurality of captured imagesbelonging to the captured image group. For example, the optimum sectiondetermination unit 280 identifies circles 1 representing the shape ofthe embryo S, circles 2 representing the shape of the first cell S1among the two cells included in the embryo S, and circles 3 representingthe shape of the second cell S2 among the two cells in association witheach other in a plurality of photographic images A, B, and C belongingto the captured image group. FIG. 36 is a diagram illustrating acorrespondence between identical elements in the plurality ofphotographic images A, B, and C belonging to the captured image group.The photographic images A, B, and C are used in FIG. 36; however, all ofthe plurality of photographic images belonging to the captured imagegroup may be used. At that time, the optimum section determination unit280 may identify the circles 1, 2, and 3 by using a condition that thecircle 1 including the circles 2 and 3 does not cross the circles 2 and3. Further, the optimum section determination unit 280 may identify thecircles 1, 2, and 3 using a condition that the identical circles havesubstantially the same size and the positions of the identical circlesare shifted based on a rule in the plurality of photographic images.

Referring to FIGS. 35 and 36, the photographic image A is an imagecaptured when an illuminator 101 ga located at one end of theilluminator group G is used for illumination. The photographic image Bis an image captured when an illuminator 101 gb located at the other endof the illuminator group G is used for illumination. The photographicimage C is an image captured when an illuminator 101 gc located at thecenter of the illuminator group G is used for illumination. Accordingly,the positions of the circles 1, 2, and 3 are shifted based on a rulefrom the photographic image A to the photographic image B.

In step S5600, the optimum section determination unit 280 calculates thepositions of the centers, which is feature points of the circles 1, 2,and 3 identified in the photographic images A, B, and C, as illustratedin FIG. 37. The optimum section determination unit 280 may determine, asthe position of the center of each circle, a point that is located atsubstantially the same distance from a plurality of points on thecircumference of the identified circle. FIG. 37 is a diagramillustrating the positions of the centers of the respective elements inthe photographic images A, B, and C. The positions of the centers of thecircles 1, 2, and 3 correspond to positions of corresponding pixels onthe light-receiving surface of the image sensor 102. Accordingly, theoptimum section determination unit 280 may calculate relative positionsof the centers of the circles 1, 2, and 3 in the photographic images A,B, and C. Here, each of the center of the circle 2 representing thefirst cell S1 and the center of the circle 3 representing the secondcell S2 is an example of a feature point of a second object in aphotographic image. However, the feature point of the second object inthe photographic image is not limited to the center of the circle. Ifthe second object does not have a spherical shape unlike the cells S1and S2, a point whose position can be identified, such as the centroid,the incenter, the circumcenter, the orthocenter, an apex, or a cornerpoint may be used as the feature point. For example, when the secondobject has a polygonal shape, a feature point such as the centroid, theincenter, the circumcenter, the orthocenter, or an apex is set for thesecond object in the photographic images. Through the processing ofsteps S5500 and S5600, feature points of the embryo S, the cell S1, andthe cell S2 in the photographic images are associated with each other.In this example, feature points of the embryo S, the cell S1, and thecell S2 are set in each photographic image after the embryo S, the cellS1, and the cell S2 in the photographic images are associated with eachother; however, the configuration is not limited to this one. The embryoS, the cell S1, and the cell S2 and feature points of the embryo S, thecell S1, and the cell S2 may be associated with each other in thephotographic images after the feature points of the embryo S, the cellS1, and the cell S2 are set in the photographic images.

In step S5700, the optimum section determination unit 280 calculatesdistances between the center positions of the same circle in thephotographic images A, B, and C. That is, the optimum sectiondetermination unit 280 calculates a positional relationship between thecenters of the same circle in the photographic images A, B, and C.Specifically, as illustrated in FIG. 38, the optimum sectiondetermination unit 280 calculates distances between the center positionsof the circle 1 in the photographic images A, B, and C, that is, shiftamounts of the center positions of the circle 1 in the photographicimages A, B, and C. Likewise, the optimum section determination unit 280calculates distances between the center positions of the circle 2 in thephotographic images A, B, and C. Further, the optimum sectiondetermination unit 280 calculates distances between the center positionsof the circle 3 in the photographic images A, B, and C. FIG. 38 is adiagram illustrating the positional relationship between the centers ofeach element in the photographic images A, B, and C. The distancesbetween the center positions of the same circle in the photographicimages A, B, and C indicate distances between three parallel lines eachpassing through the center of the circle. In FIG. 38, the distancesbetween the center positions of the same circle in the photographicimages A, B, and C are the distances along a direction in which theilluminators 101 belonging to the illuminator group G are arranged.

In step S5800, the optimum section determination unit 280 calculates theposition of the center of the sphere that represents the shape of eachof the embryo S, the first cell S1, and the second cell S2 from thedistances between the center positions of the corresponding circle inthe photographic images A, B, and C and the positions of theilluminators 101 corresponding to the respective photographic images A,B, and C. For example, as illustrated in FIG. 39, a center Ca of thecircle 1 in the photographic image A, a center Cb of the circle 1 in thephotographic image B, and a center Cc of the circle 1 in thephotographic image C are arranged on the light-receiving surface of theimage sensor 102 in accordance with a distance D1 between the center Caand the center Cc and a distance D2 between the center Cc and the centerCb. FIG. 39 is a diagram illustrating a positional relationship betweenthe centers of an element in the photographic images A, B, and C and thecorresponding illuminators 101.

In this case, the centers Ca, Cb, and Cc can be arranged substantiallyin a line. The position of the illuminator used for capturing thephotographic image A, that is, the position of the illuminator 101 gacorresponding to the photographic image A, and the center Ca areconnected by a line L1. Similarly, the position of the illuminator 101gb corresponding to the photographic image B and the center Cb areconnected by a line L3. The position of the illuminator 101 gccorresponding to the photographic image C and the center Cc areconnected by a line L2. The optimum section determination unit 280determines the position of an intersection point C1 of the lines L1, L2,and L3 as the position of the three-dimensional center of the embryo S.The height of the intersection point C1 of the lines L1, L2, and L3 fromthe light-receiving surface of the image sensor 102 is the height of thecenter of the embryo S. If the lines L1, L2, and L3 do not cross at asingle point, the position at which the sum of distances to the linesL1, L2, and L3 is the smallest may be set as the center position of theembryo S. The optimum section determination unit 280 also performs thesimilar processing for the circles 2 and 3 and calculates the centerpositions of the first cell S1 and the second cell S2.

In step S5900, the optimum section determination unit 280 determines aplane including the center of the first cell S1 and the center of thesecond cell S2 as the optimum focal plane, that is, the optimum section.For example, as illustrated in FIG. 40, the three-dimensional positionof the center C2 of the first cell S1 is calculated based on thepositions of the three centers of the circle 2 representing the firstcells S1 in the photographic images A, B, and C and the positions of theilluminators 101 ga, 101 gb, and 101 gc in step S5800. Similarly, thethree-dimensional position of the center C3 of the second cell S2 iscalculated based on the three center positions of the circle 3representing the second cell S2 in the photographic images A, B, and Cand the positions of the illuminators 101 ga, 101 gb, and 101 gc. FIG.40 is a diagram illustrating a positional relationship between thecenters C2 and C3 of the two cells S1 and S2 included in the embryo S.

The optimum section determination unit 280 determines a plane thatpasses through the centers C2 and C3 as the optimum focal plane. At thattime, the optimum section determination unit 280 may determine theoptimum focal plane so that the optimum focal plane has a tilt closer toa tilt of a plane that is parallel to the light-receiving surface of theimage sensor 102 from among planes that pass through the centers C2 andC3. Specifically, the optimum section determination unit 280 maydetermine the optimum focal plane so that the optimum focal plane has atilt angle of 10 degree or less with respect to a plane that is parallelto the light-receiving surface of the image sensor 102.

When the embryo S includes three or more cells, the optimum sectiondetermination unit 280 may determine the optimum focal plane so that theoptimum focal plane includes a largest number of centers of the cellsfor which the position of the center is derived by performing processingsimilar to the above one. In addition to or separately from the abovecondition, the optimum section determination unit 280 may determine theoptimum focal plane so that distances from the centers of all of thecells to the optimum focal plane become the smallest, that is, the sumof the distances from the centers of all of the cells becomes thesmallest. In addition to or separately from the above condition, theoptimum section determination unit 280 may determine the optimum focalplane so that the optimum focal plane passes through the centers ofthree cells among the cells for which the position of the center issuccessfully determined by performing processing similar to the aboveone and distances from the centers of the other cells to the optimumfocal plane become the smallest. Further, the optimum sectiondetermination unit 280 may determine the optimum focal plane so that theoptimum focal plane has a tilt closer to a tilt of a plane that isparallel to the light-receiving surface of the image sensor 102 duringthe determination.

In addition, when there are two or more candidate optimum focal planesas illustrated in FIG. 33, the optimum plane determination unit 280 maycalculate two optimum focal planes and employ both or one of the twooptimum focal planes. In addition, regarding display of the optimumsectional image at the optimum focal plane described below, optimumsectional images at two optimum focal planes may be displayed or anoptimum sectional image at one of the two optimum focal planes may bedisplayed.

In step S6000, the optimum section determination unit 280 sends positioninformation of the determined optimum focal plane to the refocusingprocessing unit 220. The refocusing processing unit 220 performs therefocusing process by using the photographic images stored in thestorage unit 120 on the basis of the plurality of photographic images,the position information regarding the plurality of illuminators 101,and the position information regarding the optimum focal plane anddetermines pixel values of pixels at the optimum focal plane. Therefocusing processing unit 220 may determine pixel values of the pixelsat the optimum focal plane by using information included in thereference sectional images Ic at the respective reference focal planesFP as in the sectional image generation operation performed by thesectional image generation unit 270 in the first embodiment.

In step S6100, the optimum sectional image generation unit 290generates, as an in-focus image, an optimum sectional image which isimage data that can be output to a display or the like, on the basis ofthe result of the refocusing processing performed in step S6000, anddisplays the optimum sectional image on the display unit 150. FIGS. 41and 42 illustrate an example of the optimum sectional image displayed onthe display screen 151 of the display unit 150. FIG. 41 is a diagramillustrating an example in which the optimum sectional image of theembryo S is displayed on the display screen 151 of the display unit 150.FIG. 42 is a diagram illustrating an example in which the optimumsectional image of the embryo S is displayed using a photograph. FIG. 42illustrates an example of the embryo S including four cells.

In step S6200, a focal plane change icon 158 is displayed on the displayscreen 151 together with an optimum sectional image Io of the embryo S.When the user of the image generation system 10A displays anothersectional image of the embryo S, the user selects the focal plane changeicon 158 via the CG operation input unit 140 to execute the associatedinstruction (yes in step S6200). If an instruction to change thedisplayed section from the optimum sectional image Io is not issued viathe CG operation input unit 140 (no in step S6200), the optimumsectional image Io is kept displayed.

In step S6300, in response to executing an instruction associated withthe focal plane change icon 158, the image operation unit 260 of theimage generation device 200A displays a focal plane selection screen,such as ones illustrated in FIGS. 43 and 44, on the display screen 151of the display unit 150. FIG. 43 is a diagram illustrating an example ofthe focal plane selection screen displayed on the display screen 151 ofthe display unit 150. FIG. 44 is a diagram illustrating an example inwhich the focal plane selection screen is displayed by using aphotograph. FIG. 44 illustrates an example in which the embryo Sincludes four cells.

As in the first embodiment, the 3D model A of the embryo S defined usingthe xyz space, the horizontal movement scroll bar 152, the verticalmovement scroll bar 153, the section movement scroll bar 154, themovement position display portion 155, the section position displayportion 156, and the refocus icon 157 are displayed in the focal planeselection screen. The reference axis C that passes through the center ofthe 3D model A and the sectional plane B are displayed together with the3D model A. The reference axis C can be rotated with respect to thecenter of the 3D model A, and the sectional plane B can be slid alongthe reference axis C via an operation of the CG operation input unit140. Note that the 3D model A may be configured to rotate in a givenmanner and the sectional plane B may be configured to slide along thefixed reference axis C. In either case, the sectional plane B canrepresent a given section of the 3D model A. A detailed sectional image159 of the embryo S at the sectional plane B is shown in the focal planeselection screen.

In step S6400, in response to execution of an instruction associatedwith the refocus icon 157 via an operation of the CG operation inputunit 140 (yes in step S6400), the image operation unit 260 of the imagegeneration device 200A determines the position of the displayedsectional plane B as the position of a section for which the detailedsectional image of the embryo S is to be displayed. The refocusingprocessing unit 220 and the in-focus image generation unit 230 of theimage generation device 200A generate a detailed sectional image of theembryo S, which is an in-focus image obtained using the sectional planeB as the focal plane, and display the detailed sectional image on thedisplay screen 151 (step S6500). The detailed sectional image of theembryo S may be generated in a manner similar to that performed by thesectional image generation unit 270 in the first embodiment; however,the detailed sectional image of the embryo S may be generated byperforming the refocusing process by using photographic images stored inthe storage unit 120 and using the sectional plane B as the focal plane.In addition, when an instruction associated with the refocus icon 157 isnot executed (no in step S6400), the sectional image of the embryo S iskept displayed on the display screen 151 of the display unit 150.

The 3D model of the embryo S may be generated in a manner similar tothat performed by the 3D image generation unit 250 in the firstembodiment.

The following configuration is another example of the method forgenerating the 3D model A of the embryo S in the second embodiment.

The 3D image generation unit 250 obtains information regarding theposition of the center C1 of the embryo S, the position of the center C2of the first cell S1, and the position of the center C3 of the secondcell S2 that are determined by the optimum section determination unit280. The 3D image generation unit 250 further obtains, from the storageunit 120, photographic images obtained under illumination by theilluminators 101 that are located directly above or at positions closestto positions directly above the centers C1, C2, and C3 in a directionperpendicular to the light-receiving surface of the image sensor 102.Then, the 3D image generation unit 250 extracts an outline having thehighest contrast, specifically, a circular outline, from each of thephotographic images.

The outline extracted from a photographic image corresponding toillumination performed by the illuminator 101 located directly above orat a position closest to a position directly above the center C1corresponds to an outline of a section that passes through the center ofthe embryo S, that is, the two-dimensional outline of the planar shapewhen the embryo S is viewed in a direction perpendicular to thelight-receiving surface of the image sensor 102. In this way, the 3Dimage generation unit 250 identifies the spherical shape of the embryoS.

Likewise, the outline extracted from a photographic image correspondingto illumination performed by the illuminator 101 located directly aboveor at a position closest to a position directly above the center C2corresponds to an outline of a section that passes through the center ofthe first cell S1, that is, the two-dimensional outline of the planarshape when the first cell S1 is viewed in a direction perpendicular tothe light-receiving surface of the image sensor 102. In this way, the 3Dimage generation unit 250 identifies the spherical shape of the firstcell S1.

The outline extracted from a photographic image corresponding toillumination performed by the illuminator 101 located directly above orat a position closest to a position directly above the center C3corresponds to an outline of a section that passes through the center ofthe second cell S2, that is, the two-dimensional outline of the planarshape when the second cell S2 is viewed in a direction perpendicular tothe light-receiving surface of the image sensor 102. In this way, the 3Dimage generation unit 250 identifies the spherical shape of the secondcell S2.

Accordingly, the 3D image generation unit 250 generates the 3D model Aof the embryo S on the basis of the center positions and the sphericalshapes of the embryo S, the first cell S1, and the second cell S2. Atthat time, the 3D image generation unit 250 is capable of display the 3Dmodel A of the embryo S in a translucent or transparent state in whicharrangement of the first cell S1 and the second cell S2 can be checked.

2-3. Advantageous Effects

As described above, the image generation device 200A according to thesecond embodiment selects a section that includes the largest number ofcenters of cells, which are feature points of the embryo S that is anobject to be imaged, and displays an in-focus image of the selectedsection. The displayed in-focus image of the section can show manyfeatures of the embryo S. Thus, the image generation device 200A iscapable of automatically generating and providing useful information tothe user.

First Modification of Embodiments

A first modification of the first and second embodiments will bedescribed next. The first modification is a modification regarding therefocusing process performed by the image generation system. In thefirst and second embodiments described above, the illumination positionis selected in step S1350 illustrated in FIG. 10; however, in the firstmodification, an image is selected. The first modification will bedescribed below in terms of differences from the first and secondembodiments.

FIG. 45 is a flowchart illustrating an example of an operation of therefocusing processing unit 220 according to the first modification ofthe embodiments. In FIG. 45, steps S1341 to S1390 are performed insteadof steps S1340 to S1390 illustrated in FIG. 10. In FIG. 45, steps thatare substantially the same as those of FIG. 10 are denoted by the samereference signs, and a description thereof is omitted appropriately.

In FIG. 45, if all images are used in addition processing (yes in stepS1341), the process returns to step S1320. On the other hand, if any ofthe images is not used in addition processing (no in step S1341), theprocess proceeds to step S1351. The refocusing processing unit 220selects one of the images stored in the storage unit 120 (step S1351).In this step, an image that has not been used in addition processing isselected.

The refocusing processing unit 220 obtains, from the storage unit 120,illumination position information corresponding to the image selected instep S1351 (step S1359). The processing to be performed thereafter issubstantially the same as that illustrated in FIG. 10 except that theoperation of obtaining an image in step S1370 is omitted.

As described above, according to the refocusing processing method of theimage generation system according to the first modification, pixelvalues of a plurality of images that correspond to each pixel of anin-focus image can be used for the pixel of the in-focus image as in thefirst and second embodiments even if selection of an illuminationposition is replaced with selection of an image. Consequently, ahigh-quality in-focus image of an object is successfully generated.

Second Modification of Embodiments

A second modification of the first and second embodiments will bedescribed next. The second modification is a modification regarding therefocusing process performed by the image generation system. In thefirst and second embodiments described above, pixels of an in-focusimage are sequentially selected in steps S1320 and S1330 of FIG. 10;however, pixels of a captured image are sequentially selected in thesecond modification. That is, the second modification differs from thefirst and second embodiments in that a pixel of the captured image isselected first and then a point on the focal plane corresponding to theselected pixel is determined. The pixel value of the selected pixel ofthe captured image is reflected for a pixel of an in-focus image thatcorresponds to the point on the focal plane thus determined. The secondmodification will be described below in terms of differences from thefirst and second embodiments.

FIG. 46 is a flowchart illustrating an example of an operation of therefocusing processing unit 220 according to the second modification ofthe embodiments. In FIG. 46, steps that are substantially the same asthose of FIG. 10 are denoted by the same reference signs, and adescription thereof is omitted appropriately.

In step S1322, the refocusing processing unit 220 determines whether therefocusing processing has been finished for all the images captured instep S1100. The refocusing processing indicates processing from stepS1322 to step S1392. If the refocusing processing has been finished forall the images (yes in step S1322), the process proceeds to step S1400.If the refocusing processing has not been finished for any of the imagescaptured in step S1100 (no in step S1322), the process proceeds to stepS1332.

In step S1332, the refocusing processing unit 220 selects a capturedimage from among the images captured in step S1100 and stored in thestorage unit 120. A captured image selected in this step is an imagethat has not been subjected to the refocusing processing. Hereinafter,the image selected in step S1332 is referred to as a selected image.

In step S1333, the refocusing processing unit 220 obtains illuminationposition information corresponding to the selected image. For example,the refocusing processing unit 220 obtains illumination positioninformation with reference to correspondences each between an image andillumination position information illustrated in FIG. 6. Here, thedescription will be given of the case where the position informationregarding the illuminator 101 a is obtained.

In step S1342, the refocusing processing unit 220 determines whetheraddition processing has been finished for all pixels of the selectedimage. If addition processing has been finished for all pixels of theselected image (yes in step S1342), the refocusing processing unit 220finishes the addition processing, and the process returns to step S1322.On the other hand, if addition processing has not been finished for anyof the pixels of the selected image (no in step S1342), the processproceeds to step S1352. The addition processing indicates processingfrom step S1342 to step S1392.

In step S1352, the refocusing processing unit 220 selects a pixel of theselected image. A pixel selected in this step is a pixel that has notbeen subjected to addition processing. FIG. 47 illustrates a pluralityof points 1302 a to 1302 e on the light-receiving surface thatcorrespond to a plurality of pixels included in the selected image. Adescription will be given of the case where a pixel corresponding to thepoint 1302 a on the light-receiving surface is selected from theselected image as illustrated in FIG. 48. Hereinafter, the pixelselected in step S1352 is also referred to as an addition-target pixel.

In step S1372, the refocusing processing unit 220 calculates a positionof an intersection point 1303 a at which the focal plane 1100 crosses aline connecting the point 1302 a on the light-receiving surface and theposition of the illuminator 101 a as illustrated in FIG. 48.Hereinafter, the intersection point 1303 a is also referred to as anaddition-target point.

In step S1382, the refocusing processing unit 220 calculates a pixelvalue of the addition-target pixel of the selected image thatcorresponds to the point 1302 a on the light-receiving surface that isto be added to pixel value(s) of one or more pixels of the in-focusimage that correspond to the addition-target point (intersection point1303 a) on the focal plane.

For example, if the position of the intersection point 1303 a matchesnone of pixels (integer pixels) of the in-focus image, the refocusingprocessing unit 220 calculates a pixel value to be added for a pluralityof pixels that are adjacent to the intersection point 1303 a in thein-focus image. Specifically, the refocusing processing unit 220determines the position in the in-focus image that corresponds to theaddition-target point (intersection point 1303 a) on the focal planecalculated in step S1372 on the basis of the arrangement of pixels ofthe in-focus image.

For example, a position surrounded by four pixels (pixels A to D) isdetermined to be the position of the addition-target point in thein-focus image. In this case, the refocusing processing unit 220determines distances between the addition-target point and therespective pixels (pixels A to D) that are adjacent to theaddition-target point in the in-focus image. The refocusing processingunit 220 calculates a pixel value to be added to each of the pixelsadjacent to the addition-target point by using the calculated distancesand the pixel value of the addition-target pixel. For example, therefocusing processing unit 220 calculates a pixel value to be added toeach of the pixels so that a pixel whose distance to the addition-targetpoint is relatively large in the in-focus image has a relatively largepixel value. Specifically, the refocusing processing unit 220 calculatesa pixel value La to be added to the pixel A by using Equation 5 below,for example.

$\begin{matrix}{{La} = \frac{a \times L}{a + b + c + d}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

In Equation 5, a denotes a distance between the pixel A and theaddition-target point in the in-focus image, b denotes a distancebetween the pixel B and the addition-target point in the in-focus image,c denotes a distance between the pixel C and the addition-target pointin the in-focus image, and d denotes a distance between the pixel D andthe addition-target pixel in the in-focus image. In addition, L denotesa pixel value of the addition-target pixel of the captured image.

In step S1392, the refocusing processing unit 220 adds the pixel valuecalculated in step S1382 to pixel value(s) of one or more pixels of thein-focus image.

As a result of iteration of processing form step S1342 to step S1392,the pixel values of all the pixels of the selected image aresuccessfully reflected to pixel values of the respective pixels of thein-focus image.

Further, addition processing is performed for all the pixels of thecaptured image as a result of iteration of processing from step S1322 tostep S1392, and consequently the in-focus image at the focal plane issuccessfully generated.

A specific example of the respective steps of the flowchart of FIG. 46will be described with reference to FIG. 50. The description will begiven of the case where the image sensor 102 and the focal plane satisfythe following conditions. The length of a longer side (i.e., a sideparallel to the x axis) of the light-receiving surface of the imagesensor 10 is 6 mm, whereas the length of a shorter side (i.e., a sideparallel to the y axis) of the light-receiving surface is 4 mm. Thefocal plane is tilted with respect to the x axis by 30 degrees and withrespect to the y axis by 0 degrees. An area of the focal plane is equalto an area of the light-receiving surface of the image sensor 102. Thatis, the focal plane is a rectangular plane of 6 mm×4 mm. One of theshorter sides of the focal plane extends in parallel to the y axis onthe y-z plane as illustrated in FIG. 50, whereas the other shorter sideof the focal plane extends in parallel to the y axis on the x-y planewith the x coordinate fixed to the position of approximately 5.2 mm. Thecoordinates (x, y, z) of the center of the focal plane are equal to(2.6, 2, 1.5).

It is assumed that an image is selected in step S1332, an illuminationposition (7.5, 2, 10) corresponding to the image is obtained in stepS1333, and an addition-target pixel (1.7, 2, 0) is selected in stepS1352. In this case, coordinates (2.6, 2, 1.5) of the addition-targetpoint, which is an intersection point of the focal plane and a line thatpasses through the addition-target pixel (1.7, 2, 0) and theillumination position (7.5, 2.0, 10), are calculated. Then, the pixelvalue of the addition-target pixel is distributed and added to the pixelvalues of the pixels adjacent to the addition-target point in thein-focus image in step S1382.

As described above, according to the second modification, a pixel valueof a first pixel of a captured image is applicable to pixel value(s) ofone or more second pixels of an in-focus image that correspond to aposition of an intersection point of a focal plane and a line connectingthe position of the first pixel on the light-receiving surface of theimage sensor 102 and the position of the illuminator. Accordingly, thepixel value of each pixel of the captured image is successfullyreflected in a pixel value of the corresponding pixel of an in-focusimage, and consequently, a high-quality in-focus image of an object issuccessfully generated.

Third Modification of Embodiments

A third modification of the first and second embodiments will bedescribed next. The third modification is a modification regarding theimage generation system. In the third modification, the illuminationposition is adaptively determined in accordance with a determined focalplane and an object is imaged by using an illuminator located at thedetermined illumination position, which is different from the first andsecond embodiments. The third modification will be described below interms of differences from the first and second embodiments.

Configuration of Image Generation System

FIG. 51 is a block diagram illustrating the functional configuration ofan image generation system 10B according to the third modification. InFIG. 51, components that are substantially the same as those of FIG. 1are denoted by the same reference signs, and a description thereof isomitted appropriately. The image generation system 10B includes theimaging device 100, an image generation device 200B, the storage unit120, and the display unit 150.

Configuration of Imaging Device

The imaging device 100 includes the plurality of illuminators 101, theimage sensor 102 that records, for each pixel, intensity of light, andthe imaging control unit 103.

The imaging control unit 103 controls operations of the plurality ofilluminators 101 and the image sensor 102 in accordance withillumination range information obtained from an illumination rangedetermination unit 300 (described later). Specifically, the imagingcontrol unit 103 causes the plurality of illuminators 101 located atdifferent positions to sequentially radiate light. The imaging controlunit 103 causes the image sensor 102 to capture an image of an objectevery time the object is irradiated with light by one of the pluralityof illuminators 101.

Configuration of Image Generation Device

The image generation device 200B includes the focal plane determinationunit 210, the illumination range determination unit 300, the refocusingprocessing unit 220, the in-focus image generation unit 230, the targetobject extraction unit 240, the 3D image generation unit 250, the imageoperation unit 260, and the sectional image generation unit 270.

The illumination range determination unit 300 determines theillumination position corresponding to the focal plane determined by thefocal plane determination unit 210. A specific example of how theillumination position is determined will be described with reference toFIGS. 52 and 53.

FIG. 52 is an explanatory diagram schematically illustrating a methodfor determining the illumination position in accordance with the thirdmodification. FIG. 53 is a schematic diagram illustrating acorrespondence between a relationship between the focal length of a lensand the depth of field and a relationship between arrangement of a pointlight source at the time of refocusing and the depth of field. FIG.53(a) illustrates a relationship between the focal length of a lens andthe depth of field, whereas FIG. 53(b) illustrates a relationshipbetween arrangement of a point light source at the time of refocusingand the depth of field.

Referring to FIG. 53, f denotes the focal length of a lens, s denotes adistance to a subject, and t denotes a distance from the lens to animage plane. In addition, F denotes an F-number, ε denotes ½ of a depthof focus, and δ denotes the diameter of the permissible circle ofconfusion. Further, Sn denotes a near-point distance, Sf denotes afar-point distance, Dn denotes a near depth of field, and Df denotes afar depth of field.

The depth of field achieved by refocusing is determined in accordancewith an extent of a illumination position distribution range. Referringto FIG. 53(b), the illumination position distribution range denoted by adot line corresponds to the diameter of the lens in FIG. 53(a). In thecase of the lens illustrated in FIG. 53(a), light reflected by thesurface of a subject passes through the lens and forms an image on thefocal plane. The depth of field is the sum of the near depth of field Dnand the far depth of field Df. Since refocusing is performed for imagingbased on transmitted light in the present disclosure, the focal planecorresponds to the position of the subject illustrated in FIG. 53(a). InFIG. 53(b), the image sensor is located on the left side of the focalplane. In the third modification, the depth of field can be calculatedby setting a pixel pitch of the image sensor as the permissible circleof confusion although nothing is located on the right side of the arrayof point light sources in FIG. 53(b).

For example, the range of the illumination position that is necessary togenerate an in-focus image at the image plane illustrated in FIG. 52corresponds to the size of the lens placed in parallel to the focalplane as illustrated in FIG. 53. The range of the illumination positionof the case where observation of a subject located at the focal pointrequires a lens having a diameter of 10 mm that is apart from thesubject by 5 mm is represented by the following circle. That is, therange of the illumination position is represented by a circle that isparallel to the focal plane, is apart from the focal plane by 5 mm, hasthe center at the intersection point of the normal that passes throughthe center of the focal plane and a plane parallel to the focal plane,and has a diameter of 10 mm. The position of the illuminator arranged ina region obtained by mapping this range of the illumination positiononto a flat or curved surface on which the point light source isactually arranged (e.g., the curved surface illustrated in FIG. 3 or theflat surface illustrated in FIG. 4) is the illumination position that isdetermined to be suitable for the focal plane by the focal planedetermination unit 210.

Operation of Refocusing Process of Image Generation System

An operation of a refocusing process of the image generation system 10Bthus configured will be described next. FIG. 54 is a flowchartillustrating an example of the operation of the refocusing process ofthe image generation system 10B according to the third modification. InFIG. 54, steps that are substantially the same as those of FIG. 7 aredenoted by the same reference signs, and a description thereof isomitted appropriately.

As illustrated in FIG. 54, first, the focal plane determination unit 210determines a focal plane (step S1200).

The illumination range determination unit 300 determines a range of theillumination position corresponding to the focal plane determined instep S1200 (step S2200).

The imaging device 100 irradiates an object with light by sequentiallyusing illuminators corresponding to the range of the illuminationposition determined in step S2200 among the plurality of illuminators101. The imaging device 100 records intensity of light that has receivedat the light-receiving surface of the image sensor 102 every time theobject is irradiated with light by the corresponding illuminator tocapture an image of the object. The captured image is stored in thestorage unit 120 together with the position information of theilluminator used to irradiate the object with light during imaging (stepS2300). Specifically, the imaging control unit 103 of the imaging device100 selects two or more illuminators located in the determined range ofthe illumination position from among the plurality of illuminators 101on the basis of the range of the illumination position determined instep S2200. The imaging control unit 103 then causes the two or moreselected illuminators to sequentially irradiate the object with light ina predetermined order and causes the image sensor 102 to image theobject. The imaging device 100 captures images of the object by usingthe illuminators located in the determined range of the illuminationposition by iterating irradiation of the object with light and imagingof the object. Since the following operation is substantially the sameas that illustrated in FIG. 7 according to the first embodiment, adescription thereof is omitted.

As described above, the image generation system according to the thirdmodification is capable of determining a range of the illuminationposition on the basis of information regarding the focal plane andsequentially irradiating an object with light by using illuminatorscorresponding to the determined range of the illumination position.Accordingly, the object is successfully imaged by using illuminatorssuitable for generation of an in-focus image at the focal plane, and thetime for imaging is successfully reduced.

Other Embodiments

While the image generation systems according to one or a plurality ofaspects have been described on the basis of the embodiments and themodifications of the embodiments, the present disclosure is not limitedto these embodiments and modifications. Embodiments achieved by applyingvarious modifications conceived by a person skilled in the art to theembodiments and modifications and embodiments achieved by using elementsof different embodiments in combination may also be within the scope ofthe one or plurality of aspects as long as these embodiments do notdepart of the essence of the present disclosure.

The present disclosure can be widely employed in devices that generatean image of cultured cells or a cell mass, such as an embryo, and areuseful when an object is imaged in an incubator.

What is claimed is:
 1. An image generation device comprising: a plurality of illuminators; an image sensor having a surface on which an object is placed; and a control circuit that generates an in-focus image of the object at a virtual focal plane located between the image sensor and the plurality of illuminators, wherein the object includes a first object and one or more second objects included in the first object, and wherein the control circuit (a1) obtains a plurality of images captured by the image sensor, each of the plurality of images being captured when a corresponding one of the plurality of illuminators irradiates the object with light, (a2) identifies feature points of the one or more second objects included in each of the plurality of images that have been obtained, (a3) calculates three-dimensional positions of the feature points of the one or more second objects on the basis of positions of the feature points of the one or more second objects in each of the plurality of images and positions of the plurality of illuminators, and (a4) determines a section of the first object including a largest number of feature points of second objects among the one or more second objects, generates an in-focus image of the section, and causes the in-focus image of the section to be displayed on a display screen.
 2. The image generation device according to claim 1, wherein the control circuit associates with each other the feature points of each of the one or more second objects in the plurality of images when the feature point of the second object is identified.
 3. The image generation device according to claim 1, wherein the first object is a spherical embryo, wherein each of the one or more second objects is a cell, and wherein the feature point is a center point of the cell.
 4. An image generation method for generating an image of an object placed on an image sensor by using an in-focus image at a virtual focal plane located between the image sensor and a plurality of illuminators, the object including a first object and one or more second objects included in the first object, the image generation method comprising: (b1) obtaining a plurality of images captured by sequentially causing the plurality of illuminators to irradiate the object with light; (b2) identifies feature points of the one or more second objects included in each of the plurality of images that have been obtained; (b3) calculating three-dimensional positions of the feature points of the one or more second objects on the basis of positions of the feature points of the one or more second objects in the plurality of images and positions of the plurality of illuminators; and (b4) determining a section of the first object including a largest number of feature points of second objects among the one or more second objects, generating an in-focus image of the section, and causing the in-focus image of the section to be displayed on a display screen, at least one of (b1) to (b4) being performed by a control circuit.
 5. The image generation method according to claim 4, wherein the feature points of each of the one or more second objects in the plurality of images are associated with each other when the feature point of the second object is identified.
 6. The image generation method according to claim 4, wherein the first object is a spherical embryo, wherein each of the one or more second objects is a cell, and wherein the feature point is a center point of the cell.
 7. A non-transitory recording medium storing a control program causing a device including a processor to perform a process, the recording medium being nonvolatile and computer-readable, the process comprising: (c1) obtaining a plurality of images, each of the plurality of images being captured by an image sensor when a corresponding one of a plurality of illuminators irradiates an object with light, the object being placed on the image sensor and including a first object and one or more second objects included in the first object; (c2) identifying feature points of the one or more second objects in each of the plurality of images that have been obtained; (c3) calculating three-dimensional positions of the feature points of the one or more second objects on the basis of positions of the feature points of the one or more second objects in the plurality of images and positions of the plurality of illuminators; and (c4) determining a section of the first object including a largest number of feature points of second objects among the one or more second objects, generating an in-focus image by using the section as a virtual focal plane, and causing the in-focus image of the section to be displayed on a display screen.
 8. The non-transitory recording medium according to claim 7, wherein the feature points of each of the one or more second objects in the plurality of captured images are associated with each other when the feature point of the second object is identified.
 9. The non-transitory recording medium according to claim 7, wherein the first object is a spherical embryo, wherein each of the one or more second objects is a cell, and wherein the feature point is a center point of the cell.
 10. A processor-implemented method comprising: causing a first illuminator to irradiate an object including a first cell and a second cell with light and causing an image sensor to image the object during the irradiation by the first illuminator to capture a first image; causing a second illuminator to irradiate the object with light after the irradiation by the first illuminator and causing the image sensor to image the object during the irradiation by the second illuminator to capture a second image; detecting first edges in the first image; determining a first outline of the first cell in the first image and a second outline of the second cell in the first image, by using (i) edges included in the first edges, (ii) a predetermined first radius of the first cell or a predetermined first diameter of the first cell, and (iii) a predetermined second radius of the second cell or a predetermined second diameter of the second cell; determining a first center of the first outline and a second center of the second outline; detecting second edges in the second image; determining a third outline of the first cell in the second image and a fourth outline of the second cell in the second image, by using (i) edges included in the second edges, (ii) the predetermined first radius or the predetermined first diameter, and (iii) the predetermined second radius or the predetermined second diameter; determining a third center of the third outline and a fourth center of the fourth outline; determining a first line including a first point and a second point, the first point being included in a location of a pixel, in the image sensor, providing a pixel value of the first center in the first image, the second point being included in a location of the first illuminator; determining a second line including a third point and a fourth point, the third point being included in a location of a pixel, in the image sensor, providing a pixel value of the third center in the second image, the second point being included in a location of the second illuminator; determining a first intersection of the first line and the second line; determining a third line including a fifth point and the second point, the fifth point being included in a location of a pixel, in the image sensor, providing a pixel value of the second center in the first image; determining a fourth line including a sixth point and the fourth point, the sixth point being included in a location of a pixel, in the image sensor, providing a pixel value of the fourth center in the second image; determining a second intersection of the third line and the fourth line; determining a plane including the first intersection and the second intersection; and causing a display to display an image based on an image on the plane. 